YARN中的分布式缓存功能

问题描述：

当前，我正在使用MAP-REDUCE YARN框架.并在伪分布式模式下使用hadoop. 我想在这里使用分布式缓存"功能来添加一些要缓存的文件，并在我的地图功能中使用它.我该如何做到这一点.

Currently i am using MAP-REDUCE YARN framework. And using hadoop in pseudo distributed mode. I want to use "Distributed Cache" feature here to add some files to cache and use it in my map function. How can i achieve this.

答

如何将文件添加到分布式缓存:

使用hadoop选项

hadoop jar <application jar> <main class> <input> <output> -files <absolute path to distributed cache file>

使用分布式缓存API:

job.addCacheFile(uri);

hadoop -files选项或分布式缓存API将缓存文件复制到所有任务节点，并使其在执行期间可用于mapper/reduce.

hadoop -files option or Distributed cache API copies the cache files to all the task nodes and make it available for mapper/ reducer during execution.

如何访问分布式缓存:

重写Mapper/reducer中的设置方法，并从上下文中调用getCacheFiles. 下面的示例代码:

Override setup method in Mapper/ reducer and call getCacheFiles from context. Sample code below:

    @Override
    protected void setup(Context context)
            throws IOException, InterruptedException {

        Path[] localPaths = context.getCacheFiles();
        if (localPaths.length == 0) {
            throw new FileNotFoundException("Distributed cache file not found.");
        }
        File localFile = new File(localPaths[0].toString());
        // code to process cache file

    }

context.getCacheFiles方法返回在配置"中设置的文件的URI数组.

context.getCacheFiles method returns an URI array of the files set in the Configuration.

YARN中的分布式缓存功能

相关推荐