k8s v1.9.9 二进制部署 （6）所有节点安装flannel网络插件和docker
1.二进制包安装flannel
$cd /root/
https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
$tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
$ll
$mkdir /usr/libexec/flannel/
$cp mk-docker-opts.sh /usr/libexec/flannel/
$cp flanneld /usr/bin
2.配置和启动fannel
2.1 配置文件 /usr/lib/systemd/system/flanneld.service
注：
这一步建议vim编辑，我用eof方式写入时，发现vim将变量内容当成变量读取，结果为空，导致丢失部分内容。
$vim /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/flanneld
-etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS}
-etcd-prefix=${FLANNEL_ETCD_PREFIX}
$FLANNEL_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
2.2 配置文件 /etc/sysconfig/flanneld
注：红色内容写入etcd集群IP地址。
$cat > /etc/sysconfig/flanneld <<eof
# Flanneld configuration options
# etcd url location. Point this to the server where etcd runs
# etcd config key. This is the configuration key that flannel queries
# For address range assignment
# Any additional options that you want to pass
eof
2.3 在etcd中创建网络配置（仅在master配置）
注：红色内容写入etcd集群IP地址。
etcdctl --endpoints=https://192.168.7.131:2379,https://192.168.7.132:2379,https://192.168.7.133:2379
--ca-file=/etc/kubernetes/ssl/ca.pem
--cert-file=/etc/kubernetes/ssl/kubernetes.pem
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem
mkdir /kube-centos/network
etcdctl --endpoints=https://192.168.7.131:2379,https://192.168.7.132:2379,https://192.168.7.133:2379
--ca-file=/etc/kubernetes/ssl/ca.pem
--cert-file=/etc/kubernetes/ssl/kubernetes.pem
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem
3.启动flannel
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl status flanneld
4.验证
注: 这时三个节点上查看ip，都能看到生成了一块flannel网卡，IP都是172.18.0.0/16地址段的。
$ip a
任意节点运行
$etcdctl --endpoints=${ETCD_ENDPOINTS}
--ca-file=/etc/kubernetes/ssl/ca.pem
--cert-file=/etc/kubernetes/ssl/kubernetes.pem
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem
ls /kube-centos/network/subnets
$etcdctl --endpoints=${ETCD_ENDPOINTS}
--ca-file=/etc/kubernetes/ssl/ca.pem
--cert-file=/etc/kubernetes/ssl/kubernetes.pem
--key-file=/etc/kubernetes/ssl/kubernetes-key.pem
get /kube-centos/network/config
5.安装docker
5.1.上传docker文件夹到三个节点上
注：这里离线部署的，可以yum部署建议版本18或者19开头的。
(未测试)二进制部署docker地址如下，下载解压后，将docker二进制执行文件放到/usr/bin/下，继续后面的配置，应该没问题的。
https://download.docker.com/linux/static/stable/x86_64/docker-19.03.1.tgz
$cd /root/
$上传docker文件夹，用winscp工具上传的。
$tar zcvf docker.tar.gz docker/
$scp docker.tar.gz root@192.168.7.132:/root
$scp docker.tar.gz root@192.168.7.133:/root
5.2.安装
#/etc/yum.repos.d/CentOS-Base.repo 需要有，部分离线包需要更新。
$cd /root/docker/
$yum -y localinstall lvm2/*.rpm container-selinux/*.rpm device-mapper-persistent-data/*.rpm
$yum -y localinstall docker-ce-cli/*.rpm
5.3.查看版本
docker --version
6. 配置文件 /usr/lib/systemd/system/docker.service
$cd /root/
$vim /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd --exec-opt native.cgroupdriver=systemd
$OPTIONS
$DOCKER_STORAGE_OPTIONS
$DOCKER_NETWORK_OPTIONS
$ADD_REGISTRY
$BLOCK_REGISTRY
$REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
7.启动docker：
systemctl daemon-reload
systemctl enable docker
systemctl start docker
systemctl status docker
8.验证docker
所有节点都生成了一块docker0网卡，且地址与flannel同网段。