用gradle构建一个简单的MapReduce项目:Hadoop依赖项没有Mapper和Reducer
我正在尝试构建一个简单的Hadoop mapreduce程序,为此我选择了Java.我检查了周围的示例代码并尝试构建自己.我创建了以下gradle脚本,当我查看已安装的依赖项时,没有一个具有Mapper或Reducer.甚至没有org.apache.hadoop.mapreduce包.
I'm trying to build a simple Hadoop mapreduce program and I chose Java for that job. I checked out the example codes around and tried to build myself. I created the following gradle script and when I looked at the installed dependencies, none had Mapper or Reducer. Not even org.apache.hadoop.mapreduce package.
group 'org.ardilgulez.demoprojects'
version '1.0-SNAPSHOT'
apply plugin: 'java'
repositories {
mavenCentral()
}
dependencies {
testCompile group: 'junit', name: 'junit', version: '4.11'
compile group: 'org.apache.hadoop', name: 'hadoop-common', version:'2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-hdfs', version: '2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-yarn-common', version: '2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-minicluster', version: '2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-core', version:'2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-jobclient', version: '2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-app', version: '2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-shuffle', version: '2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-common', version: '2.7.3'
compile group: 'org.apache.hadoop', name: 'hadoop-client', version: '2.7.3'
}
我知道在这10个hadoop依赖项中我至少不需要7个,但是我不知道其中哪个依赖项具有org.apache.hadoop.mapreduce包(我知道这11个都没有但是).
I know I'm not going to be needing at least 7 out of those 10 hadoop dependencies, but I don't know which of these dependencies have org.apache.hadoop.mapreduce package (I know these 11 don't though).
我应该添加哪个依存关系/存储库,以便实际上可以构建mapreduce作业?
Which dependency/repository should I add so that I can actually build a mapreduce job?
我可以使用原始的org.apache.hadoop软件包而不是供应商软件包(例如Cloudera)来做到这一点吗?
Can I do that with raw org.apache.hadoop packages and not vendor packages (such as Cloudera)?
提前感谢所有帮助.
这应该是正确的依赖项:
This should be the correct dependency:
compile 'org.apache.hadoop:hadoop-mapreduce-client-core:2.7.3'
compile 'org.apache.hadoop:hadoop-mapreduce-client-core:2.7.3'
请确保刷新您的gradle
项目.