Hadoop 集群-完全分布式模式(Fully-Distributed Mode)
分类:
IT文章
•
2023-12-14 14:57:19
Hadoop 集群-完全分布式模式(Fully-Distributed Mode)
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.查看官方文档
1>.查看对应Apache Hadoop版本的官方文档
博主推荐阅读:
http://hadoop.apache.org/docs/r2.10.0/
2>.查看Hadoop设置单节点群集的文档
博主推荐阅读:
http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/ClusterSetup.html
3>.Hadoop配置文件说明
Hadoop配置文件分两类：默认配置文件和自定义配置文件，只有用户想修改某一默认配置值时，才需要修改自定义配置文件，更改相应属性值。
默认配置文件：
core-default.xml:
默认配置文件存放在"${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-common/"路径下。
博主推荐阅读:
http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/core-default.xml
hdfs-default.xml:
默认配置文件存放在"${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/"路径下。
博主推荐阅读:
http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
yarn-default.xml:
默认配置文件存放在"/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/"路径下。
博主推荐阅读:
http://hadoop.apache.org/docs/r2.10.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
mapred-default.xml：
默认配置文件存放在""路径下。
博主推荐阅读:
http://hadoop.apache.org/docs/r2.10.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
自定义配置文件：
core-site.xml,hdfs-site.xml,yarn-site.xml,mapred-site.xml这四个配置文件存放默认存放在"$HADOOP_HOME/etc/hadoop"这个路径下。
用户可以根据项目需求重新进行修改配置，自定义的值会覆盖上面对应默认配置文件的值哟。
博主推荐阅读:
http://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml
-rw-r--r-- 1 root root 104408 Mar 12 03:27 /yinzhengjie/softwares/hadoop-2.10.0/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml | wc -l
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
-rw-r--r-- 1 root root 160516 Mar 12 03:27 /yinzhengjie/softwares/hadoop-2.10.0/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml | wc -l
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
-rw-r--r-- 1 root root 132138 Mar 12 03:27 /yinzhengjie/softwares/hadoop-2.10.0/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml | wc -l
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
-rw-r--r-- 1 root root 74650 Mar 12 03:27 /yinzhengjie/softwares/hadoop-2.10.0/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml | wc -l
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll ${HADOOP_HOME}/share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
二.完全分布式准备环境
1>.实验环境说明
本篇博客采用6台Linux发行版本为"CentOS Linux release 7.6.1810 (Core)"操作系统部署Hadoop完全分布式集群。
本篇博客介绍了如何安装和配置Hadoop集群，范围从几个节点到具有数千个节点的超大型集群。要使用Hadoop，你可能需要将其安装在单台机器上，可参考我之前的笔记，链接如下:
https://www.cnblogs.com/yinzhengjie2020/p/12422758.html
各节点Hadoop角色分配如下,当然你也可以将下面的角色自定义进行合并到一台主机，虽然博主不建议你这样干，但是如果你的物理机实在内存不足建议至少保留2台虚拟机，将下面的所有的角色分配到2个节点，但是对你的体验可能较差。
hadoop101.yinzhengjie.org.cn:
部署NameNode角色
hadoop102.yinzhengjie.org.cn:
部署DataNode,NodeManager角色
hadoop103.yinzhengjie.org.cn:
部署DataNode,NodeManager角色
hadoop104.yinzhengjie.org.cn:
部署DataNode,NodeManager角色
hadoop105.yinzhengjie.org.cn:
部署SecondNameNode
hadoop106.yinzhengjie.org.cn:
部署ResourceManager角色
2>.配置集群时间同步
温馨提示:
身为运维人员，只要你搭建的服务是以集群的方式工作，那第一件事情就是应该让所有节点的时间保持一致。
如果服务不依赖时间同步，如果你没有配置集群时间同步那倒无所谓，比如Hadoop服务如果各节点相差10分钟还是可以运行的，但是这对开发以后排错会产生困扰，因此我强烈建议大家配置集群时间同步。
不仅如此，因为在使用集群搭建后期的HBase，Kudu等服务，若时间不同会直接导致你集群无法启动哟~
博主推荐阅读:
https://www.cnblogs.com/yinzhengjie/p/12292549.html
3>.hadoop101.yinzhengjie.org.cn节点配置和集群各个节点实现免密登录(为管理hdfs集群和ansible环境做准备)
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#Hadoop cluster
172.200.4.101 hadoop101.yinzhengjie.org.cn
172.200.4.102 hadoop102.yinzhengjie.org.cn
172.200.4.103 hadoop103.yinzhengjie.org.cn
172.200.4.104 hadoop104.yinzhengjie.org.cn
172.200.4.105 hadoop105.yinzhengjie.org.cn
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat /etc/hosts
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa #快速生成私钥和公钥
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:+xTXq08c66RwzI70WB+HXCmPY1ml2f/g4UfAEp8We/g root@hadoop101.yinzhengjie.org.cn
The key's randomart image is:
+---[RSA 2048]----+
| |
| . . |
| + =.|
| ..O++|
| S . .+=*o|
| . = o**E|
| . + ==%oo|
| + O.X.=o|
| + =.*..|
+----[SHA256]-----+
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll ~/.ssh/
total 8
-rw------- 1 root root 1679 Mar 11 18:00 id_rsa #私钥
-rw-r--r-- 1 root root 415 Mar 11 18:00 id_rsa.pub #公钥
[root@hadoop101.yinzhengjie.org.cn ~]#
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop101.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop101.yinzhengjie.org.cn (172.200.4.101)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop101.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop101.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop101.yinzhengjie.org.cn
Last login: Wed Mar 11 17:58:12 2020 from 172.200.0.1
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
root pts/1 2020-03-11 18:04 (172.200.4.101)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop101.yinzhengjie.org.cn closed.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop101.yinzhengjie.org.cn
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop102.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop102.yinzhengjie.org.cn (172.200.4.102)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop102.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop102.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop102.yinzhengjie.org.cn
Last login: Wed Mar 11 13:01:52 2020 from 172.200.0.1
[root@hadoop102.yinzhengjie.org.cn ~]#
[root@hadoop102.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 18:06 (172.200.4.101)
[root@hadoop102.yinzhengjie.org.cn ~]#
[root@hadoop102.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop102.yinzhengjie.org.cn closed.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop102.yinzhengjie.org.cn
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop103.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop103.yinzhengjie.org.cn (172.200.4.103)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop103.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop103.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop103.yinzhengjie.org.cn
Last login: Tue Mar 10 06:37:08 2020 from 172.200.0.1
[root@hadoop103.yinzhengjie.org.cn ~]#
[root@hadoop103.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 18:10 (172.200.4.101)
[root@hadoop103.yinzhengjie.org.cn ~]#
[root@hadoop103.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop103.yinzhengjie.org.cn closed.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop103.yinzhengjie.org.cn
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop104.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop104.yinzhengjie.org.cn (172.200.4.104)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop104.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop104.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop104.yinzhengjie.org.cn
Last login: Tue Mar 10 06:37:30 2020 from 172.200.0.1
[root@hadoop104.yinzhengjie.org.cn ~]#
[root@hadoop104.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 18:11 (172.200.4.101)
[root@hadoop104.yinzhengjie.org.cn ~]#
[root@hadoop104.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop104.yinzhengjie.org.cn closed.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop104.yinzhengjie.org.cn
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop105.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop105.yinzhengjie.org.cn (172.200.4.105)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop105.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop105.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop105.yinzhengjie.org.cn
Last login: Tue Mar 10 06:37:53 2020 from 172.200.0.1
[root@hadoop105.yinzhengjie.org.cn ~]#
[root@hadoop105.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 18:12 (172.200.4.101)
[root@hadoop105.yinzhengjie.org.cn ~]#
[root@hadoop105.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop105.yinzhengjie.org.cn closed.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop105.yinzhengjie.org.cn
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop106.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop106.yinzhengjie.org.cn (172.200.4.106)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop106.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop106.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
root pts/1 2020-03-11 20:13 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh root@hadoop106.yinzhengjie.org.cn
Last login: Wed Mar 11 21:13:39 2020 from 172.200.0.1
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 21:13 (172.200.0.1)
root pts/1 2020-03-11 21:14 (172.200.4.101)
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop106.yinzhengjie.org.cn closed.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-11 17:58 (172.200.0.1)
root pts/1 2020-03-11 20:13 (172.200.0.1)
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop106.yinzhengjie.org.cn
4>.hadoop106.yinzhengjie.org.cn节点仅和yarn集群配置免密码登录
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:8KEEDjqqHLpGPK9HOz/j88WHLZDXXGLYKqTApi8HN7Q root@hadoop106.yinzhengjie.org.cn
The key's randomart image is:
+---[RSA 2048]----+
| . . |
| ..o . o |
|o =. o... + . |
|..+ o.o+..= o |
|o+ E ..+So o |
|++=.. = o |
|+o+o. = o |
| oo= + . o |
|o.o +o=. |
+----[SHA256]-----+
[root@hadoop106.yinzhengjie.org.cn ~]#
![]()
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop102.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop102.yinzhengjie.org.cn (172.200.4.102)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop102.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop102.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 06:49 (172.200.0.1)
root pts/1 2020-03-12 07:31 (172.200.0.1)
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh root@hadoop102.yinzhengjie.org.cn
Last login: Thu Mar 12 07:35:49 2020 from 172.200.4.101
[root@hadoop102.yinzhengjie.org.cn ~]#
[root@hadoop102.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 07:41 (172.200.4.106)
[root@hadoop102.yinzhengjie.org.cn ~]#
[root@hadoop102.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop102.yinzhengjie.org.cn closed.
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 06:49 (172.200.0.1)
root pts/1 2020-03-12 07:31 (172.200.0.1)
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop102.yinzhengjie.org.cn
![]()
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop103.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop103.yinzhengjie.org.cn (172.200.4.103)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop103.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop103.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 06:49 (172.200.0.1)
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh root@hadoop103.yinzhengjie.org.cn
Last login: Thu Mar 12 07:20:02 2020 from 172.200.0.1
[root@hadoop103.yinzhengjie.org.cn ~]#
[root@hadoop103.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 07:20 (172.200.0.1)
root pts/1 2020-03-12 07:28 (172.200.4.106)
[root@hadoop103.yinzhengjie.org.cn ~]#
[root@hadoop103.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop103.yinzhengjie.org.cn closed.
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop103.yinzhengjie.org.cn
![]()
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop104.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop104.yinzhengjie.org.cn (172.200.4.104)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop104.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop104.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 06:49 (172.200.0.1)
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh root@hadoop104.yinzhengjie.org.cn
Last login: Thu Mar 12 06:55:07 2020 from 172.200.4.101
[root@hadoop104.yinzhengjie.org.cn ~]#
[root@hadoop104.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 07:29 (172.200.4.106)
[root@hadoop104.yinzhengjie.org.cn ~]#
[root@hadoop104.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop104.yinzhengjie.org.cn closed.
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop104.yinzhengjie.org.cn
![]()
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop105.yinzhengjie.org.cn
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hadoop105.yinzhengjie.org.cn (172.200.4.105)' can't be established.
ECDSA key fingerprint is SHA256:y6iS5ipSyWSGRmgcjivbWhd78pKfrcuQHeBPd5H9/U8.
ECDSA key fingerprint is MD5:da:0f:2a:93:c0:d4:6e:7e:13:16:61:f1:93:a7:38:01.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hadoop105.yinzhengjie.org.cn's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@hadoop105.yinzhengjie.org.cn'"
and check to make sure that only the key(s) you wanted were added.
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 06:49 (172.200.0.1)
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh root@hadoop105.yinzhengjie.org.cn
Last login: Thu Mar 12 06:55:07 2020 from 172.200.4.101
[root@hadoop105.yinzhengjie.org.cn ~]#
[root@hadoop105.yinzhengjie.org.cn ~]# who
root pts/0 2020-03-12 07:29 (172.200.4.106)
[root@hadoop105.yinzhengjie.org.cn ~]#
[root@hadoop105.yinzhengjie.org.cn ~]# exit
logout
Connection to hadoop105.yinzhengjie.org.cn closed.
[root@hadoop106.yinzhengjie.org.cn ~]#
[root@hadoop106.yinzhengjie.org.cn ~]# ssh-copy-id root@hadoop105.yinzhengjie.org.cn
5>.安装ansible并配置hadoop主机清单
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# yum list ansible
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Determining fastest mirrors
* base: mirror.bit.edu.cn
* extras: mirror.bit.edu.cn
* updates: mirrors.aliyun.com
Available Packages
ansible.noarch 2.4.2.0-2.el7 extras
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# yum list ansible
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# yum -y install ansible
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.bit.edu.cn
* extras: mirror.bit.edu.cn
* updates: mirrors.aliyun.com
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/2): extras/7/x86_64/primary_db | 164 kB 00:00:00
(2/2): updates/7/x86_64/primary_db | 6.7 MB 00:00:01
Resolving Dependencies
--> Running transaction check
---> Package ansible.noarch 0:2.4.2.0-2.el7 will be installed
--> Processing Dependency: sshpass for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: python2-jmespath for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: python-six for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: python-setuptools for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: python-passlib for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: python-paramiko for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: python-jinja2 for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: python-httplib2 for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: python-cryptography for package: ansible-2.4.2.0-2.el7.noarch
--> Processing Dependency: PyYAML for package: ansible-2.4.2.0-2.el7.noarch
--> Running transaction check
---> Package PyYAML.x86_64 0:3.10-11.el7 will be installed
--> Processing Dependency: libyaml-0.so.2()(64bit) for package: PyYAML-3.10-11.el7.x86_64
---> Package python-httplib2.noarch 0:0.9.2-1.el7 will be installed
---> Package python-jinja2.noarch 0:2.7.2-4.el7 will be installed
--> Processing Dependency: python-babel >= 0.8 for package: python-jinja2-2.7.2-4.el7.noarch
--> Processing Dependency: python-markupsafe for package: python-jinja2-2.7.2-4.el7.noarch
---> Package python-paramiko.noarch 0:2.1.1-9.el7 will be installed
--> Processing Dependency: python2-pyasn1 for package: python-paramiko-2.1.1-9.el7.noarch
---> Package python-passlib.noarch 0:1.6.5-2.el7 will be installed
---> Package python-setuptools.noarch 0:0.9.8-7.el7 will be installed
--> Processing Dependency: python-backports-ssl_match_hostname for package: python-setuptools-0.9.8-7.el7.noarch
---> Package python-six.noarch 0:1.9.0-2.el7 will be installed
---> Package python2-cryptography.x86_64 0:1.7.2-2.el7 will be installed
--> Processing Dependency: python-idna >= 2.0 for package: python2-cryptography-1.7.2-2.el7.x86_64
--> Processing Dependency: python-cffi >= 1.4.1 for package: python2-cryptography-1.7.2-2.el7.x86_64
--> Processing Dependency: python-ipaddress for package: python2-cryptography-1.7.2-2.el7.x86_64
--> Processing Dependency: python-enum34 for package: python2-cryptography-1.7.2-2.el7.x86_64
---> Package python2-jmespath.noarch 0:0.9.0-3.el7 will be installed
---> Package sshpass.x86_64 0:1.06-2.el7 will be installed
--> Running transaction check
---> Package libyaml.x86_64 0:0.1.4-11.el7_0 will be installed
---> Package python-babel.noarch 0:0.9.6-8.el7 will be installed
---> Package python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 will be installed
--> Processing Dependency: python-backports for package: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
---> Package python-cffi.x86_64 0:1.6.0-5.el7 will be installed
--> Processing Dependency: python-pycparser for package: python-cffi-1.6.0-5.el7.x86_64
---> Package python-enum34.noarch 0:1.0.4-1.el7 will be installed
---> Package python-idna.noarch 0:2.4-1.el7 will be installed
---> Package python-ipaddress.noarch 0:1.0.16-2.el7 will be installed
---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed
---> Package python2-pyasn1.noarch 0:0.1.9-7.el7 will be installed
--> Running transaction check
---> Package python-backports.x86_64 0:1.0-8.el7 will be installed
---> Package python-pycparser.noarch 0:2.14-1.el7 will be installed
--> Processing Dependency: python-ply for package: python-pycparser-2.14-1.el7.noarch
--> Running transaction check
---> Package python-ply.noarch 0:3.4-11.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================================================================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================================================================================================================================
Installing:
ansible noarch 2.4.2.0-2.el7 extras 7.6 M
Installing for dependencies:
PyYAML x86_64 3.10-11.el7 base 153 k
libyaml x86_64 0.1.4-11.el7_0 base 55 k
python-babel noarch 0.9.6-8.el7 base 1.4 M
python-backports x86_64 1.0-8.el7 base 5.8 k
python-backports-ssl_match_hostname noarch 3.5.0.1-1.el7 base 13 k
python-cffi x86_64 1.6.0-5.el7 base 218 k
python-enum34 noarch 1.0.4-1.el7 base 52 k
python-httplib2 noarch 0.9.2-1.el7 extras 115 k
python-idna noarch 2.4-1.el7 base 94 k
python-ipaddress noarch 1.0.16-2.el7 base 34 k
python-jinja2 noarch 2.7.2-4.el7 base 519 k
python-markupsafe x86_64 0.11-10.el7 base 25 k
python-paramiko noarch 2.1.1-9.el7 base 269 k
python-passlib noarch 1.6.5-2.el7 extras 488 k
python-ply noarch 3.4-11.el7 base 123 k
python-pycparser noarch 2.14-1.el7 base 104 k
python-setuptools noarch 0.9.8-7.el7 base 397 k
python-six noarch 1.9.0-2.el7 base 29 k
python2-cryptography x86_64 1.7.2-2.el7 base 502 k
python2-jmespath noarch 0.9.0-3.el7 extras 39 k
python2-pyasn1 noarch 0.1.9-7.el7 base 100 k
sshpass x86_64 1.06-2.el7 extras 21 k
Transaction Summary
==============================================================================================================================================================================================================================================================================
Install 1 Package (+22 Dependent packages)
Total download size: 12 M
Installed size: 60 M
Downloading packages:
(1/23): PyYAML-3.10-11.el7.x86_64.rpm | 153 kB 00:00:00
(2/23): libyaml-0.1.4-11.el7_0.x86_64.rpm | 55 kB 00:00:00
(3/23): python-backports-1.0-8.el7.x86_64.rpm | 5.8 kB 00:00:00
(4/23): python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch.rpm | 13 kB 00:00:00
(5/23): python-cffi-1.6.0-5.el7.x86_64.rpm | 218 kB 00:00:00
(6/23): python-enum34-1.0.4-1.el7.noarch.rpm | 52 kB 00:00:00
(7/23): python-idna-2.4-1.el7.noarch.rpm | 94 kB 00:00:00
(8/23): python-ipaddress-1.0.16-2.el7.noarch.rpm | 34 kB 00:00:00
(9/23): python-babel-0.9.6-8.el7.noarch.rpm | 1.4 MB 00:00:00
(10/23): python-markupsafe-0.11-10.el7.x86_64.rpm | 25 kB 00:00:00
(11/23): python-httplib2-0.9.2-1.el7.noarch.rpm | 115 kB 00:00:00
(12/23): python-jinja2-2.7.2-4.el7.noarch.rpm | 519 kB 00:00:00
(13/23): python-ply-3.4-11.el7.noarch.rpm | 123 kB 00:00:00
(14/23): python-paramiko-2.1.1-9.el7.noarch.rpm | 269 kB 00:00:00
(15/23): python-pycparser-2.14-1.el7.noarch.rpm | 104 kB 00:00:00
(16/23): python-six-1.9.0-2.el7.noarch.rpm | 29 kB 00:00:00
(17/23): python-passlib-1.6.5-2.el7.noarch.rpm | 488 kB 00:00:00
(18/23): python-setuptools-0.9.8-7.el7.noarch.rpm | 397 kB 00:00:00
(19/23): python2-jmespath-0.9.0-3.el7.noarch.rpm | 39 kB 00:00:00
(20/23): sshpass-1.06-2.el7.x86_64.rpm | 21 kB 00:00:00
(21/23): python2-pyasn1-0.1.9-7.el7.noarch.rpm | 100 kB 00:00:00
(22/23): python2-cryptography-1.7.2-2.el7.x86_64.rpm | 502 kB 00:00:00
(23/23): ansible-2.4.2.0-2.el7.noarch.rpm | 7.6 MB 00:00:01
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 7.8 MB/s | 12 MB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : python2-pyasn1-0.1.9-7.el7.noarch 1/23
Installing : python-ipaddress-1.0.16-2.el7.noarch 2/23
Installing : python-six-1.9.0-2.el7.noarch 3/23
Installing : python-httplib2-0.9.2-1.el7.noarch 4/23
Installing : sshpass-1.06-2.el7.x86_64 5/23
Installing : libyaml-0.1.4-11.el7_0.x86_64 6/23
Installing : PyYAML-3.10-11.el7.x86_64 7/23
Installing : python-backports-1.0-8.el7.x86_64 8/23
Installing : python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch 9/23
Installing : python-setuptools-0.9.8-7.el7.noarch 10/23
Installing : python-babel-0.9.6-8.el7.noarch 11/23
Installing : python-passlib-1.6.5-2.el7.noarch 12/23
Installing : python-ply-3.4-11.el7.noarch 13/23
Installing : python-pycparser-2.14-1.el7.noarch 14/23
Installing : python-cffi-1.6.0-5.el7.x86_64 15/23
Installing : python-markupsafe-0.11-10.el7.x86_64 16/23
Installing : python-jinja2-2.7.2-4.el7.noarch 17/23
Installing : python-idna-2.4-1.el7.noarch 18/23
Installing : python-enum34-1.0.4-1.el7.noarch 19/23
Installing : python2-cryptography-1.7.2-2.el7.x86_64 20/23
Installing : python-paramiko-2.1.1-9.el7.noarch 21/23
Installing : python2-jmespath-0.9.0-3.el7.noarch 22/23
Installing : ansible-2.4.2.0-2.el7.noarch 23/23
Verifying : python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch 1/23
Verifying : python2-jmespath-0.9.0-3.el7.noarch 2/23
Verifying : python-enum34-1.0.4-1.el7.noarch 3/23
Verifying : python-setuptools-0.9.8-7.el7.noarch 4/23
Verifying : python-jinja2-2.7.2-4.el7.noarch 5/23
Verifying : python-six-1.9.0-2.el7.noarch 6/23
Verifying : python-idna-2.4-1.el7.noarch 7/23
Verifying : python-markupsafe-0.11-10.el7.x86_64 8/23
Verifying : python-ply-3.4-11.el7.noarch 9/23
Verifying : python-passlib-1.6.5-2.el7.noarch 10/23
Verifying : python-paramiko-2.1.1-9.el7.noarch 11/23
Verifying : python-babel-0.9.6-8.el7.noarch 12/23
Verifying : python-backports-1.0-8.el7.x86_64 13/23
Verifying : python-cffi-1.6.0-5.el7.x86_64 14/23
Verifying : python-pycparser-2.14-1.el7.noarch 15/23
Verifying : libyaml-0.1.4-11.el7_0.x86_64 16/23
Verifying : ansible-2.4.2.0-2.el7.noarch 17/23
Verifying : python-ipaddress-1.0.16-2.el7.noarch 18/23
Verifying : sshpass-1.06-2.el7.x86_64 19/23
Verifying : python-httplib2-0.9.2-1.el7.noarch 20/23
Verifying : python2-pyasn1-0.1.9-7.el7.noarch 21/23
Verifying : PyYAML-3.10-11.el7.x86_64 22/23
Verifying : python2-cryptography-1.7.2-2.el7.x86_64 23/23
Installed:
ansible.noarch 0:2.4.2.0-2.el7
Dependency Installed:
PyYAML.x86_64 0:3.10-11.el7 libyaml.x86_64 0:0.1.4-11.el7_0 python-babel.noarch 0:0.9.6-8.el7 python-backports.x86_64 0:1.0-8.el7 python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 python-cffi.x86_64 0:1.6.0-5.el7
python-enum34.noarch 0:1.0.4-1.el7 python-httplib2.noarch 0:0.9.2-1.el7 python-idna.noarch 0:2.4-1.el7 python-ipaddress.noarch 0:1.0.16-2.el7 python-jinja2.noarch 0:2.7.2-4.el7 python-markupsafe.x86_64 0:0.11-10.el7
python-paramiko.noarch 0:2.1.1-9.el7 python-passlib.noarch 0:1.6.5-2.el7 python-ply.noarch 0:3.4-11.el7 python-pycparser.noarch 0:2.14-1.el7 python-setuptools.noarch 0:0.9.8-7.el7 python-six.noarch 0:1.9.0-2.el7
python2-cryptography.x86_64 0:1.7.2-2.el7 python2-jmespath.noarch 0:0.9.0-3.el7 python2-pyasn1.noarch 0:0.1.9-7.el7 sshpass.x86_64 0:1.06-2.el7
Complete!
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# yum -y install ansible
[root@hadoop101.yinzhengjie.org.cn ~]# tail -13 /etc/ansible/hosts
#Add by yinzhengjie for hadoop hosts
#Namenode
[nn]
hadoop101.yinzhengjie.org.cn
#SecondNamenode
[2nn]
hadoop105.yinzhengjie.org.cn
#datanode
[dn]
hadoop[102:104].yinzhengjie.org.cn
#ResourceManager
[rm]
hadoop106.yinzhengjie.org.cn
[root@hadoop101.yinzhengjie.org.cn ~]#
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m ping
hadoop101.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m ping
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible 2nn -m ping
hadoop105.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible 2nn -m ping
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible dn -m ping
hadoop102.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
hadoop103.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
hadoop104.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible dn -m ping
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m ping
hadoop106.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m ping
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m ping
hadoop105.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
hadoop101.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
hadoop103.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
hadoop102.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
hadoop104.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
hadoop106.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"ping": "pong"
}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m ping
6>.使用ansible工具将"/etc/hosts"文件拷贝到其它节点
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#Hadoop cluster
172.200.4.101 hadoop101.yinzhengjie.org.cn
172.200.4.102 hadoop102.yinzhengjie.org.cn
172.200.4.103 hadoop103.yinzhengjie.org.cn
172.200.4.104 hadoop104.yinzhengjie.org.cn
172.200.4.105 hadoop105.yinzhengjie.org.cn
172.200.4.106 hadoop106.yinzhengjie.org.cn
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat /etc/hosts
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m copy -a 'src=/etc/hosts dest=/etc/hosts'
hadoop101.yinzhengjie.org.cn | SUCCESS => {
"changed": false,
"checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea",
"gid": 0,
"group": "root",
"mode": "0644",
"owner": "root",
"path": "/etc/hosts",
"size": 433,
"state": "file",
"uid": 0
}
hadoop103.yinzhengjie.org.cn | SUCCESS => {
"changed": true,
"checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea",
"dest": "/etc/hosts",
"gid": 0,
"group": "root",
"md5sum": "769e0a0d41e5be699399b271cea8a377",
"mode": "0644",
"owner": "root",
"size": 433,
"src": "/root/.ansible/tmp/ansible-tmp-1583932891.99-144884840221439/source",
"state": "file",
"uid": 0
}
hadoop102.yinzhengjie.org.cn | SUCCESS => {
"changed": true,
"checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea",
"dest": "/etc/hosts",
"gid": 0,
"group": "root",
"md5sum": "769e0a0d41e5be699399b271cea8a377",
"mode": "0644",
"owner": "root",
"size": 433,
"src": "/root/.ansible/tmp/ansible-tmp-1583932891.96-167534829416060/source",
"state": "file",
"uid": 0
}
hadoop105.yinzhengjie.org.cn | SUCCESS => {
"changed": true,
"checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea",
"dest": "/etc/hosts",
"gid": 0,
"group": "root",
"md5sum": "769e0a0d41e5be699399b271cea8a377",
"mode": "0644",
"owner": "root",
"size": 433,
"src": "/root/.ansible/tmp/ansible-tmp-1583932892.02-30034905247840/source",
"state": "file",
"uid": 0
}
hadoop104.yinzhengjie.org.cn | SUCCESS => {
"changed": true,
"checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea",
"dest": "/etc/hosts",
"gid": 0,
"group": "root",
"md5sum": "769e0a0d41e5be699399b271cea8a377",
"mode": "0644",
"owner": "root",
"size": 433,
"src": "/root/.ansible/tmp/ansible-tmp-1583932892.04-75312091987102/source",
"state": "file",
"uid": 0
}
hadoop106.yinzhengjie.org.cn | SUCCESS => {
"changed": true,
"checksum": "7aa42f0af4417ce6c188ed4cad34287dfa3570ea",
"dest": "/etc/hosts",
"gid": 0,
"group": "root",
"md5sum": "769e0a0d41e5be699399b271cea8a377",
"mode": "0644",
"owner": "root",
"size": 433,
"src": "/root/.ansible/tmp/ansible-tmp-1583932894.24-77039419077594/source",
"state": "file",
"uid": 0
}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m copy -a 'src=/etc/hosts dest=/etc/hosts'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'cat /etc/hosts'
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#Hadoop cluster
172.200.4.101 hadoop101.yinzhengjie.org.cn
172.200.4.102 hadoop102.yinzhengjie.org.cn
172.200.4.103 hadoop103.yinzhengjie.org.cn
172.200.4.104 hadoop104.yinzhengjie.org.cn
172.200.4.105 hadoop105.yinzhengjie.org.cn
172.200.4.106 hadoop106.yinzhengjie.org.cn
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#Hadoop cluster
172.200.4.101 hadoop101.yinzhengjie.org.cn
172.200.4.102 hadoop102.yinzhengjie.org.cn
172.200.4.103 hadoop103.yinzhengjie.org.cn
172.200.4.104 hadoop104.yinzhengjie.org.cn
172.200.4.105 hadoop105.yinzhengjie.org.cn
172.200.4.106 hadoop106.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#Hadoop cluster
172.200.4.101 hadoop101.yinzhengjie.org.cn
172.200.4.102 hadoop102.yinzhengjie.org.cn
172.200.4.103 hadoop103.yinzhengjie.org.cn
172.200.4.104 hadoop104.yinzhengjie.org.cn
172.200.4.105 hadoop105.yinzhengjie.org.cn
172.200.4.106 hadoop106.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#Hadoop cluster
172.200.4.101 hadoop101.yinzhengjie.org.cn
172.200.4.102 hadoop102.yinzhengjie.org.cn
172.200.4.103 hadoop103.yinzhengjie.org.cn
172.200.4.104 hadoop104.yinzhengjie.org.cn
172.200.4.105 hadoop105.yinzhengjie.org.cn
172.200.4.106 hadoop106.yinzhengjie.org.cn
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#Hadoop cluster
172.200.4.101 hadoop101.yinzhengjie.org.cn
172.200.4.102 hadoop102.yinzhengjie.org.cn
172.200.4.103 hadoop103.yinzhengjie.org.cn
172.200.4.104 hadoop104.yinzhengjie.org.cn
172.200.4.105 hadoop105.yinzhengjie.org.cn
172.200.4.106 hadoop106.yinzhengjie.org.cn
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#Hadoop cluster
172.200.4.101 hadoop101.yinzhengjie.org.cn
172.200.4.102 hadoop102.yinzhengjie.org.cn
172.200.4.103 hadoop103.yinzhengjie.org.cn
172.200.4.104 hadoop104.yinzhengjie.org.cn
172.200.4.105 hadoop105.yinzhengjie.org.cn
172.200.4.106 hadoop106.yinzhengjie.org.cn
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'cat /etc/hosts'
博主推荐阅读:
https://www.cnblogs.com/yinzhengjie/p/10447587.html
7>.使用ansible工具在所有节点批量安装rsync服务并编写集群分发脚本
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'yum -y install rsync' #安装rsync服务
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim /usr/local/bin/rsync-hadoop.sh #编写rsync同步脚本
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat /usr/local/bin/rsync-hadoop.sh
#!/bin/bash
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie
#EMAIL:y1053419035@qq.com
#判断用户是否传参
if [ $# -lt 1 ];then
echo "请输入参数";
exit
fi
#获取文件路径
file=$@
#获取子路径
filename=`basename $file`
#获取父路径
dirpath=`dirname $file`
#获取完整路径
cd $dirpath
fullpath=`pwd -P`
#同步文件到DataNode
for (( hostId=102;hostId<=106;hostId++ ))
do
#使终端变绿色
tput setaf 2
echo "******* [hadoop${hostId}.yinzhengjie.org.cn] node starts synchronizing [${file}] *******"
#使终端变回原来的颜色，即白灰色
tput setaf 7
#远程执行命令
rsync -lr $filename `whoami`@hadoop${hostId}.yinzhengjie.org.cn:${fullpath}
#判断命令是否执行成功
if [ $? == 0 ];then
echo "命令执行成功"
fi
done
[root@hadoop101.yinzhengjie.org.cn ~]#
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ll /usr/local/bin/rsync-hadoop.sh
-rw-r--r-- 1 root root 903 Mar 11 20:30 /usr/local/bin/rsync-hadoop.sh
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# chmod +x /usr/local/bin/rsync-hadoop.sh
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ll /usr/local/bin/rsync-hadoop.sh
-rwxr-xr-x 1 root root 903 Mar 11 20:30 /usr/local/bin/rsync-hadoop.sh
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# chmod +x /usr/local/bin/rsync-hadoop.sh
温馨提示:
虽然我们可以不用rsync服务，使用ansible的确是可以帮我们完成文件拷贝的，但ansible依旧时基于shutil.copy方法进行拷贝的，它并不会判断源文件和目标文件的差异，而是直接将源文件原封不动的拷贝到目标文件中。
而使用rsync做文件的复制要比ansible的copy速度快，原因是rsync只对差异文件做更新。因此生成环境中使用rsync进行同步的方式很常见，比如NFS+rsync结合使用的案例。
三.完全分布式集群配置实操案例
1>.在node101.yinzhengjie.org.cn主机安装Hadoop运行环境
此步骤只需要安装JDK，Hadoop软件并配置相应的环境变量即可。
博主推荐阅读:
https://www.cnblogs.com/yinzhengjie2020/p/12422758.html
2>.修改core-site.xml
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/core-site.xml
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop101.yinzhengjie.org.cn:9000</value>
<description>指定HDFS中NameNode的RPC地址</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/yinzhengjie/softwares/hadoop-2.10.0/data/tmp</value>
<description>指定Hadoop运行时产生文件的存储目录</description>
</property>
</configuration>
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/core-site.xml
3>.修改hdfs-site.xml
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
<description>指定HDFS副本的数量</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop105.yinzhengjie.org.cn:50090</value>
<description>指定Hadoop辅助名称节点主机配置</description>
</property>
</configuration>
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml
4>.修改yarn-site.xml
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/yarn-site.xml
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>Reducer获取数据的方式</description>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop106.yinzhengjie.org.cn</value>
<description>指定YARN的ResourceManager的地址</description>
</property>
</configuration>
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/yarn-site.xml
5>.创建mapred-site.xml文件
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# cp ${HADOOP_HOME}/etc/hadoop/mapred-site.xml.template ${HADOOP_HOME}/etc/hadoop/mapred-site.xml #需要手动创建该文件
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/mapred-site.xml
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>指定MR运行在YARN上</description>
</property>
</configuration>
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/mapred-site.xml
6>.修改hadoop-env.sh
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# grep ^export ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh | grep JAVA_HOME
export JAVA_HOME=${JAVA_HOME}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# echo ${JAVA_HOME}
/yinzhengjie/softwares/jdk1.8.0_201
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# grep ^export ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh | grep JAVA_HOME
export JAVA_HOME=/yinzhengjie/softwares/jdk1.8.0_201
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
7>.将node101.yinzhengjie.org.cn服务器的配置同步到其它节点上（可能耗时5-10分钟，如果在服务器上的话不到30秒搞定）
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'mkdir -pv /yinzhengjie/softwares/'
[WARNING]: Consider using file module with state=directory rather than running mkdir
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/’
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/’
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/’
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/’
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/’
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'mkdir -pv /yinzhengjie/softwares/' #node101.yinzhengjie.org.cn节点已经有目录不会创建，其它节点都会创建相应的目录
[root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop102.yinzhengjie.org.cn:/yinzhengjie/softwares/ #我们第一次拷贝数据使用scp命令拷贝，从第二次开始同步数据则需要使用rsync拷贝
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop103.yinzhengjie.org.cn:/yinzhengjie/softwares/
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop104.yinzhengjie.org.cn:/yinzhengjie/softwares/
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop105.yinzhengjie.org.cn:/yinzhengjie/softwares/
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop106.yinzhengjie.org.cn:/yinzhengjie/softwares/
温馨提示，以上五条命令可以简写为一条，如下所示:
[root@hadoop101.yinzhengjie.org.cn ~]# for hostId in `seq 102 106`; do scp -r /yinzhengjie/softwares/hadoop-2.10.0/ hadoop${hostId}.yinzhengjie.org.cn:/yinzhengjie/softwares/; done
拷贝完Hadoop软件包后，需要拷贝JDK的软件包，具体操作如下:
[root@hadoop101.yinzhengjie.org.cn ~]# for hostId in `seq 102 106`; do scp -r /yinzhengjie/softwares/jdk1.8.0_201/ hadoop${hostId}.yinzhengjie.org.cn:/yinzhengjie/softwares/; done
拷贝完JDK和Hadoop之后，千万别忘记拷贝我们写的环境变量文件哟,拷贝较小的文件或者同步文件的话可以考虑使用rsync脚本来拷贝(温馨提示:宿主机不建议大家对过大文件拷贝使用rsync，有可能物理操作系统程序崩溃，尤其是笔记本)，如下所示。
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /etc/profile.d/jdk.sh
******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] *******
命令执行成功
******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] *******
命令执行成功
******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] *******
命令执行成功
******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] *******
命令执行成功
******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/jdk.sh] *******
命令执行成功
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /etc/profile.d/jdk.sh
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /etc/profile.d/hadoop.sh
******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] *******
命令执行成功
******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] *******
命令执行成功
******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] *******
命令执行成功
******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] *******
命令执行成功
******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/etc/profile.d/hadoop.sh] *******
命令执行成功
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /etc/profile.d/hadoop.sh
8>.格式化namenode节点
[root@hadoop101.yinzhengjie.org.cn ~]# hdfs namenode -format
9>.启动namenode
[root@hadoop101.yinzhengjie.org.cn ~]# jps
5798 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# hadoop-daemon.sh start namenode
starting namenode, logging to /yinzhengjie/softwares/fully-mode/logs/hadoop-root-namenode-hadoop101.yinzhengjie.org.cn.out
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# jps
5908 Jps
5830 NameNode
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 172.200.4.101:9000 *:*
LISTEN 0 128 *:50070 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 128 :::22 :::*
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
10>.启动datanode
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
4271 Jps
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
4282 Jps
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
4300 Jps
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5532 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6064 Jps
5830 NameNode
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5483 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible dn -m shell -a 'hadoop-daemon.sh start datanode'
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop104.yinzhengjie.org.cn.out
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop102.yinzhengjie.org.cn.out
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop103.yinzhengjie.org.cn.out
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible dn -m shell -a 'hadoop-daemon.sh start datanode'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5915 Jps
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
4809 Jps
4682 DataNode
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
4827 Jps
4700 DataNode
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
4672 DataNode
4799 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5830 NameNode
6969 Jps
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5886 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
11>.启动Secondarynamenode
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5354 Jps
5055 DataNode
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6264 Jps
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5072 DataNode
5371 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8054 Jps
7727 NameNode
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5344 Jps
5046 DataNode
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6222 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible 2nn -m shell -a 'hadoop-daemon.sh start secondarynamenode'
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting secondarynamenode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-secondarynamenode-hadoop105.yinzhengjie.org.cn.out
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible 2nn -m shell -a 'hadoop-daemon.sh start secondarynamenode'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6418 SecondaryNameNode
6506 Jps
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5072 DataNode
5429 Jps
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5046 DataNode
5402 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8227 Jps
7727 NameNode
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5413 Jps
5055 DataNode
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6281 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
12>.启动ResourceManager节点
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5072 DataNode
5499 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8376 Jps
7727 NameNode
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5480 Jps
5055 DataNode
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6640 Jps
6418 SecondaryNameNode
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5046 DataNode
5469 Jps
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6342 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'yarn-daemon.sh start resourcemanager'
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-resourcemanager-hadoop106.yinzhengjie.org.cn.out
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'yarn-daemon.sh start resourcemanager'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5046 DataNode
5530 Jps
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5543 Jps
5055 DataNode
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8535 Jps
7727 NameNode
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5072 DataNode
5557 Jps
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6418 SecondaryNameNode
6701 Jps
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6691 Jps
6415 ResourceManager
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
13>.启动NodeManager节点
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5601 Jps
5055 DataNode
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5072 DataNode
5617 Jps
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5046 DataNode
5590 Jps
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6768 Jps
6418 SecondaryNameNode
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8758 Jps
7727 NameNode
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6749 Jps
6415 ResourceManager
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nm -m shell -a 'yarn-daemon.sh start nodemanager'
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop102.yinzhengjie.org.cn.out
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop103.yinzhengjie.org.cn.out
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop104.yinzhengjie.org.cn.out
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nm -m shell -a 'yarn-daemon.sh start nodemanager'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6418 SecondaryNameNode
6823 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8915 Jps
7727 NameNode
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5833 Jps
5674 NodeManager
5055 DataNode
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5046 DataNode
5822 Jps
5663 NodeManager
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5072 DataNode
5850 Jps
5691 NodeManager
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6806 Jps
6415 ResourceManager
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
14>.配置slaves(在hadoop101.yinzhengjie.org.cn节点通过slave文件，使用ssh命令启动slave节点的进程,因此最好配置免密登录~)
root@hadoop101.yinzhengjie.org.cn ~]# echo ${HADOOP_HOME}
/yinzhengjie/softwares/hadoop-2.10.0
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/slaves
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# cat ${HADOOP_HOME}/etc/hadoop/slaves
hadoop102.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn
[root@hadoop101.yinzhengjie.org.cn ~]#
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves
******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] *******
命令执行成功
******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] *******
命令执行成功
******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] *******
命令执行成功
******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] *******
命令执行成功
******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves] *******
命令执行成功
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh /yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'cat /yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves'
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
hadoop102.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
hadoop102.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
hadoop102.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
hadoop102.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
hadoop102.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
hadoop102.yinzhengjie.org.cn
hadoop103.yinzhengjie.org.cn
hadoop104.yinzhengjie.org.cn
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'cat /yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/slaves'
15>.停止hadoop集群
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
5351 SecondaryNameNode
6282 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
9096 Jps
5531 NameNode
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
4096 DataNode
6089 NodeManager
6267 Jps
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6146 NodeManager
6324 Jps
4089 DataNode
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6257 Jps
4084 DataNode
6079 NodeManager
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
7555 ResourceManager
7866 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m shell -a 'stop-dfs.sh'
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
Stopping namenodes on [hadoop101.yinzhengjie.org.cn]
hadoop101.yinzhengjie.org.cn: stopping namenode
hadoop102.yinzhengjie.org.cn: stopping datanode
hadoop103.yinzhengjie.org.cn: stopping datanode
hadoop104.yinzhengjie.org.cn: stopping datanode
Stopping secondary namenodes [hadoop105.yinzhengjie.org.cn]
hadoop105.yinzhengjie.org.cn: stopping secondarynamenode
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m shell -a 'stop-dfs.sh' #停止HDFS集群
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
9583 Jps
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6377 Jps
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6146 NodeManager
6414 Jps
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6357 Jps
6089 NodeManager
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6347 Jps
6079 NodeManager
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
7555 ResourceManager
7925 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'stop-yarn.sh'
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
stopping yarn daemons
stopping resourcemanager
hadoop103.yinzhengjie.org.cn: stopping nodemanager
hadoop103.yinzhengjie.org.cn: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
hadoop102.yinzhengjie.org.cn: stopping nodemanager
hadoop102.yinzhengjie.org.cn: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
hadoop104.yinzhengjie.org.cn: stopping nodemanager
hadoop104.yinzhengjie.org.cn: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'stop-yarn.sh' #停止YARN集群
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6765 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
10312 Jps
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6756 Jps
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6548 Jps
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6823 Jps
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8239 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
16>.修改脚本，将hadoop的pid文件放在指定目录中，默认放在"/tmp"目录。
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'mkdir -pv /yinzhengjie/softwares/hadoop-2.10.0/pid'
[WARNING]: Consider using file module with state=directory rather than running mkdir
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
mkdir: created directory ‘/yinzhengjie/softwares/hadoop-2.10.0/pid’
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'mkdir -pv /yinzhengjie/softwares/hadoop-2.10.0/pid'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# grep HADOOP_PID_DIR ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# grep HADOOP_PID_DIR ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
#export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_PID_DIR=/yinzhengjie/softwares/hadoop-2.10.0/pid
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# grep HADOOP_MAPRED_PID_DIR ${HADOOP_HOME}/etc/hadoop/mapred-env.sh
#export HADOOP_MAPRED_PID_DIR= # The pid files are stored. /tmp by default.
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/mapred-env.sh
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# grep HADOOP_MAPRED_PID_DIR ${HADOOP_HOME}/etc/hadoop/mapred-env.sh
#export HADOOP_MAPRED_PID_DIR= # The pid files are stored. /tmp by default.
export HADOOP_MAPRED_PID_DIR=/yinzhengjie/softwares/hadoop-2.10.0/pid
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/mapred-env.sh
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# grep YARN_PID_DIR ${HADOOP_HOME}/etc/hadoop/yarn-env.sh
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/yarn-env.sh
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# grep YARN_PID_DIR ${HADOOP_HOME}/etc/hadoop/yarn-env.sh
export YARN_PID_DIR=/yinzhengjie/softwares/hadoop-2.10.0/pid
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# vim ${HADOOP_HOME}/etc/hadoop/yarn-env.sh
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] *******
命令执行成功
******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] *******
命令执行成功
******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] *******
命令执行成功
******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] *******
命令执行成功
******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/hadoop-env.sh] *******
命令执行成功
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/mapred-env.sh
******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] *******
命令执行成功
******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] *******
命令执行成功
******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] *******
命令执行成功
******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] *******
命令执行成功
******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/mapred-env.sh] *******
命令执行成功
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/mapred-env.sh
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/yarn-env.sh
******* [hadoop102.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] *******
命令执行成功
******* [hadoop103.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] *******
命令执行成功
******* [hadoop104.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] *******
命令执行成功
******* [hadoop105.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] *******
命令执行成功
******* [hadoop106.yinzhengjie.org.cn] node starts synchronizing [/yinzhengjie/softwares/hadoop-2.10.0/etc/hadoop/yarn-env.sh] *******
命令执行成功
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# rsync-hadoop.sh ${HADOOP_HOME}/etc/hadoop/yarn-env.sh
17>.启动hadoop集群
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6879 Jps
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6943 Jps
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6886 Jps
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
10580 Jps
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6674 Jps
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8361 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m shell -a 'start-dfs.sh'
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
Starting namenodes on [hadoop101.yinzhengjie.org.cn]
hadoop101.yinzhengjie.org.cn: starting namenode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-namenode-hadoop101.yinzhengjie.org.cn.out
hadoop102.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop102.yinzhengjie.org.cn.out
hadoop104.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop104.yinzhengjie.org.cn.out
hadoop103.yinzhengjie.org.cn: starting datanode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-datanode-hadoop103.yinzhengjie.org.cn.out
Starting secondary namenodes [hadoop105.yinzhengjie.org.cn]
hadoop105.yinzhengjie.org.cn: starting secondarynamenode, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/hadoop-root-secondarynamenode-hadoop105.yinzhengjie.org.cn.out
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible nn -m shell -a 'start-dfs.sh' #启动HDFS集群
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
11121 Jps
10774 NameNode
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6987 DataNode
7118 Jps
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6923 DataNode
7054 Jps
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6806 Jps
6717 SecondaryNameNode
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6930 DataNode
7061 Jps
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8419 Jps
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'start-yarn.sh'
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
starting yarn daemons
starting resourcemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-resourcemanager-hadoop106.yinzhengjie.org.cn.out
hadoop103.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop103.yinzhengjie.org.cn.out
hadoop104.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop104.yinzhengjie.org.cn.out
hadoop102.yinzhengjie.org.cn: starting nodemanager, logging to /yinzhengjie/softwares/hadoop-2.10.0/logs/yarn-root-nodemanager-hadoop102.yinzhengjie.org.cn.out
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible rm -m shell -a 'start-yarn.sh' #启动YARN集群
![]()
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6922 Jps
6717 SecondaryNameNode
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
10774 NameNode
11383 Jps
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
7221 NodeManager
7399 Jps
6987 DataNode
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
7157 NodeManager
7335 Jps
6923 DataNode
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
6930 DataNode
7164 NodeManager
7342 Jps
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
8885 Jps
8574 ResourceManager
[root@hadoop101.yinzhengjie.org.cn ~]#
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'jps'
18>.查看hadoop集群的pid文件
[root@hadoop101.yinzhengjie.org.cn ~]# ansible all -m shell -a 'ls -l /yinzhengjie/softwares/hadoop-2.10.0/pid/'
hadoop101.yinzhengjie.org.cn | SUCCESS | rc=0 >>
total 8
-rw-r--r-- 1 root root 6 Mar 12 08:40 hadoop-root-namenode.pid
hadoop102.yinzhengjie.org.cn | SUCCESS | rc=0 >>
total 8
-rw-r--r-- 1 root root 5 Mar 12 08:40 hadoop-root-datanode.pid
-rw-r--r-- 1 root root 5 Mar 12 08:41 yarn-root-nodemanager.pid
hadoop105.yinzhengjie.org.cn | SUCCESS | rc=0 >>
total 4
-rw-r--r-- 1 root root 5 Mar 12 08:40 hadoop-root-secondarynamenode.pid
hadoop104.yinzhengjie.org.cn | SUCCESS | rc=0 >>
total 8
-rw-r--r-- 1 root root 5 Mar 12 08:40 hadoop-root-datanode.pid
-rw-r--r-- 1 root root 5 Mar 12 08:41 yarn-root-nodemanager.pid
hadoop103.yinzhengjie.org.cn | SUCCESS | rc=0 >>
total 8
-rw-r--r-- 1 root root 5 Mar 12 08:40 hadoop-root-datanode.pid
-rw-r--r-- 1 root root 5 Mar 12 08:41 yarn-root-nodemanager.pid
hadoop106.yinzhengjie.org.cn | SUCCESS | rc=0 >>
total 4
-rw-r--r-- 1 root root 5 Mar 12 08:41 yarn-root-resourcemanager.pid
[root@hadoop101.yinzhengjie.org.cn ~]#
