HDP集群部署 !/bin/bash ------------------------------------- 系统环境初始化脚本 本脚本仅支持centos， centos7.7验证成功 使用root用户运行 ------------------------------------ 1 依赖安装 2 关闭防火墙 3 关闭SELINUX 4 修改UMASK 5 关闭swappiness 6 资源限制 7 时钟同步 8 禁止huge_page 9 生成公钥私钥 !/bin/bash java.sh 依赖安装 创建目录 下载repo url文件 查看repo 下载repo文件到本地 新建本地repo 启动http服务 Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Custom JDK GPL License for LZO: https://www.gnu.org/li
一、环境准备
- centos7 安装和系统环境处理
1）运行初始化脚本
-------------------------------------
系统环境初始化脚本
本脚本仅支持centos， centos7.7验证成功
使用root用户运行
------------------------------------
1 依赖安装
yum clean all
yum makecache
yum update -y
yum install -y openssh-server vim gcc gcc-c++ glibc-headers bzip2-devel lzo-devel curl wget openssh-clients zlib-devel autoconf automake cmake libtool openssl-devel fuse-devel snappy-devel telnet unzip zip net-tools.x86_64 firewalld systemd svn ncurses-devel bzip2 psmisc libxslt-devel libxml2-devel libxslt.so.1 cyrus-sasl-plain yum groupinstall chinese-support mod_ssl ntp
2 关闭防火墙
echo "close firewall ----------------------------------------------------------"
firewall-cmd --state
systemctl status firewalld.service
systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl status firewalld.service
3 关闭SELINUX
echo "close SELINUX ----------------------------------------------------------"
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' '/etc/selinux/config'
4 修改UMASK
echo "set umask 0022 ----------------------------------------------------------"
umask 0022
echo umask 0022 >> /etc/profile
5 关闭swappiness
swap_dev=swapon -s | awk '{if (NR>1){print $1}}'
echo "swap device is ${swap_dev} ----------------------------------------------------------"
echo "close swap device ${swap_dev} ----------------------------------------------------------"swapoff ${swap_dev}
if [ "$?" != 0 ] ; then
echo "close swap device ${swap_dev} failed ----------------------------------------------------------"
else
echo "close swap device ${swap_dev} success ----------------------------------------------------------"
fi
6 资源限制
echo "set ulimit 102400 ----------------------------------------------------------"
ulimit -Hn 102400
ulimit -Sn 102400
echo "* - nproc 102400" >> /etc/security/limits.conf
echo "* - nofile 102400" >> /etc/security/limits.conf
7 时钟同步
echo "start ntp ----------------------------------------------------------"
systemctl start ntpd
systemctl enable ntpd
systemctl status ntpd
8 禁止huge_page
echo "close huge_page ----------------------------------------------------------"
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
9 生成公钥私钥
echo "ssh-keygen ----------------------------------------------------------"
mkdir ~/.ssh
ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
2）如果时区不合适，需要修改时区
[root@centos ~]# timedatectl
3）hostname 修改
[root@centos ~]# hostnamectl set-hostname hdp-master01
4）添加hosts
vim /etc/hosts
5）免密登陆
ambari-server root用户ssh免密登陆所有其他节点
[root@centos ~]# ssh-copy-id root@node03
2. java8
在 /etc/profile.d/目录下，增加java.sh
!/bin/bash
java.sh
JAVA_HOME=/usr/local/services/jdk1.8.0_191
JRE_HOME=/usr/local/services/jdk1.8.0_191/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH
3. python2.7.5
yum -y install python-pip
pip install --upgrade pip
4. mysql
Node03:3306
root/12345678
yum install mysql-connector-java*
5. 目录规划
mkdir /hadoop
将所有磁盘挂载或者链接到此目录下
/hadoop/disk01
/hadoop/disk02
二、建设本地yum repo
有网络可以直接通过网络，无网络需要先拷贝所有需要下载的文件
依赖安装
yum install yum-utils createrepo httpd
创建目录
mkdir -p /var/www/html
mkdir -p /var/www/html/ambari
mkdir -p /var/www/html/hdp
下载repo url文件
cd /etc/yum.repos.d/
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari.repo
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/hdp.repo
查看repo
yum repolist
下载repo文件到本地
cd /var/www/html/ambari
reposync -r ambari-2.7.4.0
cd /var/www/html/hdp
reposync -r HDP-3.1.4.0
reposync -r HDP-UTILS-1.1.0.22
新建本地repo
createrepo /var/www/html/ambari/ambari-2.7.4.0
createrepo /var/www/html/hdp/HDP-3.1.4.0
createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.22
启动http服务
systemctl start httpd.service
systemctl enable httpd.service
修改repo web地址
[root@node03 ~]#vim /etc/yum.repos.d/ambari.repo
[ambari-2.7.4.0]
baseurl=http://node03/ambari/ambari-2.7.4.0
...
[root@node03 ~]#vim /etc/yum.repos.d/hdp.repo
[HDP-3.1.4.0]
name=HDP Version - HDP-3.1.4.0
baseurl=http://node03/hdp/HDP-3.1.4.0
...
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://node03/hdp/HDP-UTILS-1.1.0.22
...
/etc/yum.repos.d/ambari.repo和hdp.repo需要拷贝到ambari-server将要部署的服务器
三、部署ambari-server
1.准备数据库
CREATE DATABASE ambari default character set utf8mb4 collate utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON . TO 'root'@'%' IDENTIFIED BY '12345678' WITH GRANT OPTION;
1.安装ambari-server
yum install ambari-server
2.设置
[root@node01 /etc/yum.repos.d]#ambari-server setup
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
Enter choice (1): 2
Path to JAVA_HOME: /usr/local/services/jdk1.8.0_191
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Enter advanced database configuration [y/n] (n)? y
Configuring database...
Choose one of the following options:
[3] - MySQL / MariaDB
Enter choice (1): 3
Hostname (localhost): node03
Port (3306): 3306
Database name (ambari): ambari
Username (ambari): root
Enter Database Password (bigdata):
Re-enter password: 12345678
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
3.建表
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
4.启动
端口 8080
ambari-server start
ambari-server status
ambari-server stop
ambari-server restart
四、添加组件
注册 admin/admin
创建集群
选择版本设置仓库
添加节点
上传ambari-server所在服务器的私钥id_rsa
配置用户和ssh端口
使用的host不是域名，下一步
注册所有节点
添加组件
选择组件实际部署节点
设置grafana 帐号密码以及Activity 密码
设置文件目录（需要提前规划）
设置各个组件的启动用户
HDFS服务设置
部署
五、测试
测试hdfs 上传文件
hadoop fs mkdir /tmp/test/in
hadoop fs -put text.txt /tmp/test/in
测试yarn 运行任务
hadoop jar /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /tmp/test/in /tmp/test/out