为什么使用kubeadm安装Kubenetes集群?

kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。这个工具能通过两条指令快速完成一个kubernetes集群的部署。
网上很多人说通过二进制安装能了解到配置的细节,其实通过kubeadm安装也能查看到配置的细节。可以自动生成证书,对初学者带来了不少便利。

网络环境

我们完全模拟生产环境中,不可以访问外部互联网的情况。
基础的yum源是有提供的,像什么docker-ce、kubernetes的源是没有的。
k8s.gcr.io、quay.io这些域名也是不可以访问的。

环境准备

主机名称节点类型IP地址
k8s-mastermaster节点172.16.64.233
k8s-node1node节点172.16.64.232
k8s-node2node节点172.16.64.235

master节点需要至少2个CPU,不然会报如错误:

error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

关闭selinux

SELINUX=enforcing替换成SELINUX=disabled

sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config
setenforce 0

查看一下selinux的状态。

getenforce
Permissive

关闭Swap

swapoff -a
cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak | grep -v swap > /etc/fstab

grep -v swap是查找不包含swap的行。
查看一下swap的情况,Swap已经全部为0了。

free -m
              total        used        free      shared  buff/cache   available
Mem:            972         142         715           7         114         699
Swap:             0           0           0

设置主机名

  • 在master节点上设置主机名。
hostnamectl set-hostname k8s-master
  • 在node1节点上设置主机名。
hostnamectl set-hostname k8s-node1
  • 在node2节点上设置主机名。
hostnamectl set-hostname k8s-node2
  • 在master上查看主机名。
hostname
k8s-master

设置hosts

cat >> /etc/hosts <<EOF
172.16.64.233   k8s-master
172.16.64.232   k8s-node1
172.16.64.235   k8s-node2
EOF

修改sysctl.conf

暂时未修改,装docker的时候会自动修改。可以暂时先跳过这一步。
如果未修改成功,在执行docker info命令时,会显示如下提示信息。

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
cat /proc/sys/net/bridge/bridge-nf-call-iptables
0
cat /proc/sys/net/bridge/bridge-nf-call-ip6tables
0

可通过以下方法来做修改。

# 修改 /etc/sysctl.conf
# 如果有配置,则修改
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g"  /etc/sysctl.conf
# 可能没有,追加
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
sysctl -p

也就是在/etc/sysctl.conf末尾加上如下内容:

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

同时让配置生效sysctl -p

安装Docker

  • 下载Docker

由于我们在生产环境中是没法连接互联网的,所以要提前准备好docker rpm包。
我们在另一台可以联网的机器上下载安装所需的软件。

  • 添加 Docker yum源

在联网的机器上,下载docker,配置docker-ce源。

cd /etc/yum.repos.d/
wget https://download.docker.com/linux/centos/docker-ce.repo
  • 查看Docker版本
yum list docker-ce --showduplicates
...
docker-ce.x86_64                            18.06.3.ce-3.el7                                   docker-ce-stable
docker-ce.x86_64                            3:18.09.0-3.el7                                    docker-ce-stable
...

我们选择安装docker-ce.18.06.3.ce-3.el7

  • 执行下载Docker
yum install --downloadonly --downloaddir ~/k8s/docker docker-ce-18.06.3.ce-3.el7

docker及其依赖会下载到~/docker文件夹中。

我们可以看到只有docker-ce是来自docker-ce-stable源的。

===============================================================================================================
 Package                          架构             版本                       源                          大小
===============================================================================================================
正在安装:
 docker-ce                        x86_64           18.06.3.ce-3.el7           docker-ce-stable            41 M
为依赖而安装:
 audit-libs-python                x86_64           2.8.5-4.el7                base                        76 k
 checkpolicy                      x86_64           2.5-8.el7                  base                       295 k
 container-selinux                noarch           2:2.107-3.el7              extras                      39 k
 libcgroup                        x86_64           0.41-21.el7                base                        66 k
 libseccomp                       x86_64           2.3.1-3.el7                base                        56 k
 libsemanage-python               x86_64           2.5-14.el7                 base                       113 k
 libtool-ltdl                     x86_64           2.4.2-22.el7_3             base                        49 k
 policycoreutils-python           x86_64           2.5-33.el7                 base                       457 k
 python-IPy                       noarch           0.75-6.el7                 base                        32 k
 setools-libs                     x86_64           3.3.8-4.el7                base                       620 k

所以,我们只需要把docker-ce-18.06.3.ce-3.el7.x86_64.rpm拷贝到master及node节点里面。
在master及node节点里创建~/k8s/docker目录,用于存放docker安装rpm包。

mkdir -p ~/k8s/docker
  • 拷贝到k8s集群
scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.233:~/k8s/docker/
scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.232:~/k8s/docker/
scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.235:~/k8s/docker/
  • 安装Docker
yum install k8s/docker/docker-ce-18.06.3.ce-3.el7.x86_64.rpm
  • 设置开机启动
systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

我们可以查看一下安装包到底生成了哪些文件。

rpm -ql docker-ce

或者

rpm -qpl k8s/docker/docker-ce-18.06.3.ce-3.el7.x86_64.rpm
  • 启动Docker
systemctl start docker
  • 查看Docker信息
docker info
...
Cgroup Driver: cgroupfs
...

呆会儿我们还需要修改这个值。

安装k8s组件

由于kubeadm是依赖kubelet, kubectl的,所以我们只需要下载kubeadm的rpm,其依赖就自动下载下来了。但是版本可能不是我们想要的,所以可能需要单独下载。比如我下载kubeadm-1.15.6,它依赖的可能是kubelet-1.16.x。

  • 下载k8s组件

我们需要安装kubeadm, kubelet, kubectl,版本需要一致。在可以连外网的机器上下载组件,同上面docker。

  • 添加kubernetes yum源
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enabled=1
EOF
  • 查看kubeadm版本
yum list kubeadm --showduplicates

kubeadm.x86_64                                                         1.15.6-0
  • 下载kubeadm-1.15.6
yum install --downloadonly --downloaddir ~/k8s/kubernetes kubeadm-1.15.6

根据如下依赖关系

====================================================================================================================================================
 Package                                    架构                       版本                                    源                              大小
====================================================================================================================================================
正在安装:
 kubeadm                                    x86_64                     1.15.6-0                                kubernetes                     8.9 M
为依赖而安装:
 conntrack-tools                            x86_64                     1.4.4-5.el7_7.2                         updates                        187 k
 cri-tools                                  x86_64                     1.13.0-0                                kubernetes                     5.1 M
 kubectl                                    x86_64                     1.16.3-0                                kubernetes                      10 M
 kubelet                                    x86_64                     1.16.3-0                                kubernetes                      22 M
 kubernetes-cni                             x86_64                     0.7.5-0                                 kubernetes                      10 M
 libnetfilter_cthelper                      x86_64                     1.0.0-10.el7_7.1                        updates                         18 k
 libnetfilter_cttimeout                     x86_64                     1.0.0-6.el7_7.1                         updates                         18 k
 libnetfilter_queue                         x86_64                     1.0.2-2.el7_2                           base                            23 k
 socat                                      x86_64                     1.7.3.2-2.el7                           base

我们只需要把来自kubernetes源的kubeadm和4个依赖cri-tools, kubectl, kubeletkubernetes-cni拷贝到master和node节点。

  • 下载kubelet-1.15.6
yum install --downloadonly --downloaddir ~/k8s/kubernetes kubelet-1.15.6
  • 下载kubectl-1.15.6
yum install --downloadonly --downloaddir ~/k8s/kubernetes kubectl-1.15.6
  • 拷贝到k8s集群

在master及node节点里创建~/k8s/kubernetes目录,用于存放k8s组件安装的rpm包。

mkdir -p ~/k8s/kubernetes

kubeadm

cri-tools

scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubectl

scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubelet

scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubernetes-cni

scp 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

安装k8s组件

yum install ~/k8s/kubernetes/*.rpm

这样,kubeadm, kubectl, kubelet就已经安装好了。
设置kubelet的开机启动。我们并不需要启动kubelet,就算启动,也是不能成功的。执行kubeadm命令,会生成一些配置文件 ,这时才会让kubelet启动成功的。

systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

拉取镜像

执行kubeadm时,需要用到一些镜像,我们需要提前准备。

  • 查看需要依赖哪些镜像
kubeadm config images list
W1207 18:53:23.129020   10255 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1207 18:53:23.129433   10255 version.go:99] falling back to the local client version: v1.15.6
k8s.gcr.io/kube-apiserver:v1.15.6
k8s.gcr.io/kube-controller-manager:v1.15.6
k8s.gcr.io/kube-scheduler:v1.15.6
k8s.gcr.io/kube-proxy:v1.15.6
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

在生产环境,是肯定访问不了k8s.gcr.io这个地址的。在有大陆联网的机器上,也是无法访问的。所以我们需要使用国内镜像先下载下来。

镜像地址,请参考 K8S国内的镜像源

  • 拉取镜像(在三台机器上拉取如下镜像)
docker pull gcr.azk8s.cn/google-containers/kube-apiserver:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-controller-manager:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-scheduler:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-proxy:v1.15.6
docker pull gcr.azk8s.cn/google-containers/pause:3.1
docker pull gcr.azk8s.cn/google-containers/etcd:3.3.10
docker pull gcr.azk8s.cn/google-containers/coredns:1.3.1
  • 查看拉取镜像。
docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
gcr.azk8s.cn/google-containers/kube-proxy                v1.15.6             d756327a2327        3 weeks ago         82.4MB
gcr.azk8s.cn/google-containers/kube-apiserver            v1.15.6             9f612b9e9bbf        3 weeks ago         207MB
gcr.azk8s.cn/google-containers/kube-controller-manager   v1.15.6             83ab61bd43ad        3 weeks ago         159MB
gcr.azk8s.cn/google-containers/kube-scheduler            v1.15.6             502e54938456        3 weeks ago         81.1MB
gcr.azk8s.cn/google-containers/coredns                   1.3.1               eb516548c180        10 months ago       40.3MB
gcr.azk8s.cn/google-containers/etcd                      3.3.10              2c4adeb21b4f        12 months ago       258MB
gcr.azk8s.cn/google-containers/pause                     3.1                 da86e6ba6ca1        23 months ago       742kB
  • tag镜像

为了让kubeadm程序能找到k8s.gcr.io下面的镜像,需要把刚才下载的镜像名称重新打一下tag。

docker images | grep gcr.azk8s.cn/google-containers | sed 's/gcr.azk8s.cn\/google-containers/k8s.gcr.io/' | awk '{print "docker tag " $3 " " $1 ":" $2}' | sh

删除旧的镜像,当然,你留着也不会占用太多空间。

docker images | grep gcr.azk8s.cn/google-containers | awk '{print "docker rmi " $1 ":" $2}' | sh
  • 查看镜像
docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                v1.15.6             d756327a2327        3 weeks ago         82.4MB
k8s.gcr.io/kube-apiserver            v1.15.6             9f612b9e9bbf        3 weeks ago         207MB
k8s.gcr.io/kube-controller-manager   v1.15.6             83ab61bd43ad        3 weeks ago         159MB
k8s.gcr.io/kube-scheduler            v1.15.6             502e54938456        3 weeks ago         81.1MB
k8s.gcr.io/coredns                   1.3.1               eb516548c180        10 months ago       40.3MB
k8s.gcr.io/etcd                      3.3.10              2c4adeb21b4f        12 months ago       258MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        23 months ago       742kB

镜像搞定了。

部署k8s集群

  • 初始化master节点

在master节点上执行kubeadm init命令。
我们接下来首先会使用flannel网络,所以参数中必须设置--pod-network-cidr=10.244.0.0/16,这个IP地址是固定的。

kubeadm init \
--apiserver-advertise-address=172.16.64.233 \
--pod-network-cidr=10.244.0.0/16

W1207 21:18:48.257967   10859 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1207 21:18:48.258448   10859 version.go:99] falling back to the local client version: v1.15.6
[init] Using Kubernetes version: v1.15.6
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.64.233]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [172.16.64.233 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [172.16.64.233 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 36.504799 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 1czxp7.4tt0x3lxdcus8wer
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.64.233:6443 --token 1czxp7.4tt0x3lxdcus8wer \
    --discovery-token-ca-cert-hash sha256:abb676401e56a48f07675ff802f1abedd512cce0523190b2e0f636ee6d70d8b4

解决WARNING

我们看到上面的消息中有一句

[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

还记得前面我在查看docker info时,有提到要修改cgroup driver么?现在就来修改吧。

修改或创建/etc/docker/daemon.json,添加如下内容:

{
    "exec-opts": ["native.cgroupdriver=systemd"]
}
  • 重启docker
systemctl restart docker
  • 查看修改结果,如果Cgroup Driver改为systemd后就表示成功了。
docker info
...
Cgroup Driver: systemd
...
  • 重置环境
kubeadm reset

[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W1207 22:12:18.285935   27649 reset.go:98] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get https://172.16.64.233:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: dial tcp 172.16.64.233:6443: connect: connection refused
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W1207 22:12:19.569005   27649 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
  • 再次初始化Master节点

apiserver-advertise-addresspod-network-cidr参数都可以省略掉。

kubeadm init \
--apiserver-advertise-address=172.16.64.233 \
--pod-network-cidr=10.244.0.0/16

...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.64.233:6443 --token duof19.l9q3dsh4ccen4ya0 \
    --discovery-token-ca-cert-hash sha256:66fad0ed5f46f5ea9a394276a77db16d26d60465ec9930f8c21aa924a5df9bb5

提示信息和上面初始化时的信息一样,只是少了刚才的WARNING。

按照信息提示,执行如下命令,目前登录的就是root用户,所以也不需要用sudo了。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看节点信息,节点状态为NotReady:

kubectl get no
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   2m22s   v1.15.6

往集群里面加入node节点

  • 在节点node1上,按上面的提示执行命令:
kubeadm join 172.16.64.233:6443 --token duof19.l9q3dsh4ccen4ya0 \
    --discovery-token-ca-cert-hash sha256:66fad0ed5f46f5ea9a394276a77db16d26d60465ec9930f8c21aa924a5df9bb5
    
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
  • 在Master节点上(control-plane)上查看节点信息
kubectl get no
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   NotReady   master   7m    v1.15.6
k8s-node1    NotReady   <none>   65s   v1.15.6

我们看到了多了一个节点,虽然现在都是NotReady状态。

  • Token过期后再加入节点

过了一段时间后,再加入节点,这个时候会提示token已经过期了。我们可以这样拿到token和hash值。

kubeadm token create
kubeadm token list
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

安装Network插件

  • 查看安装方法

查看flannel的官网https://github.com/coreos/flannel,找到安装方法。

For Kubernetes v1.7+ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
  • 下载yml文件

在有网络的机器上下载kube-flannel.yml文件

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

把下载好的yml文件分发到k8s集群的三台机器里面。

  • 下载镜像
cat kube-flannel.yml | grep image
        image: quay.io/coreos/flannel:v0.11.0-amd64
        ...

还记得前面方法么?不记得就回到上面再看看吧。

docker pull quay.azk8s.cn/coreos/flannel:v0.11.0-amd64
docker tag ff281650a721 quay.io/coreos/flannel:v0.11.0-amd64
docker rmi quay.azk8s.cn/coreos/flannel:v0.11.0-amd64
  • 安装flannel

我们也可以选择安装Calico网络插件。
在Master节点执行:

kubectl apply -f kube-flannel.yml

podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

查看节点信息

kubectl get no
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   33m   v1.15.6
k8s-node1    Ready    <none>   28m   v1.15.6
k8s-node2    Ready    <none>   17m   v1.15.6

这一下所有节点都已经ready了。

  • Master节点
ps -ef | grep kube
root       1652      1  3 15:13 ?        00:00:04 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1
root       1973   1909  8 15:13 ?        00:00:12 kube-apiserver --advertise-address=172.16.64.233 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root       1980   1915  1 15:13 ?        00:00:01 kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
root       1995   1936  2 15:13 ?        00:00:03 etcd --advertise-client-urls=https://172.16.64.233:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://172.16.64.233:2380 --initial-cluster=k8s-master=https://172.16.64.233:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://172.16.64.233:2379 --listen-peer-urls=https://172.16.64.233:2380 --name=k8s-master --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
root       2002   1951  2 15:13 ?        00:00:03 kube-controller-manager --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true
root       2263   2244  0 15:13 ?        00:00:00 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8s-master
root       3907   3888  0 15:14 ?        00:00:00 /usr/bin/kube-controllers
  • Worker节点
ps -ef | grep kube
root       1355      1  1 15:05 ?        00:00:06 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1
root       1669   1620  0 15:13 ?        00:00:00 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8s-node1

测试k8s集群

安装一个nginx。

  • 创建一个部署(deployment)

在master节点(Control Plane)安装一个叫nginx-deployment的deployment:

kubectl create deploy nginx-deployment --image=nginx
deployment.apps/nginx-deployment created
  • 查看deployment状态
kubectl get deploy
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   0/1     1            0           44s

发现没有READY。

  • 查看pod状态
kubectl get po
NAME                                READY   STATUS              RESTARTS   AGE
nginx-deployment-6f77f65499-htdps   0/1     ContainerCreating   0          111s

也是没有READY。继续查看详细信息。

kubectl describe po nginx-deployment-6f77f65499-htdps

...
Events:
  Type    Reason     Age    From                Message
  ----    ------     ----   ----                -------
  Normal  Scheduled  2m18s  default-scheduler   Successfully assigned default/nginx-deployment-6f77f65499-htdps to k8s-node1
  Normal  Pulling    2m17s  kubelet, k8s-node1  Pulling image "nginx"

初步判断应该是拉取镜像拉不下来,或者速度非常慢。

  • 配置docker源

在生产环境,肯定是有内部的镜像源的,在这里,我就模拟把源配置为阿里的镜像源了。
/etc/docker/daemon.json内容如下:

{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "registry-mirrors": ["http://hub-mirror.c.163.com"]
}

重启docker

systemctl restart docker

这个时候,镜像就容易拉取了。

  • 测试pod
kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-6f77f65499-htdps   1/1     Running   0          26m   10.244.1.3   k8s-node1   <none>           <none>

我们看到pod的IP为10.244.1.3。还记得我们在初始化master节点时设置的参数--pod-network-cidr=10.244.0.0/16么?

在集群内的三个节点访问nginx,能成功访问。

curl 10.244.1.3

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...
  • 创建Service

我们把deployment暴露出来。

kubectl expose deploy nginx-deployment --port=80 --type=NodePort
service/nginx-deployment exposed

查看状态

kubectl get svc
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes         ClusterIP   10.96.0.1       <none>        443/TCP        71m
nginx-deployment   NodePort    10.109.145.67   <none>        80:32538/TCP   32s

在三个节点内访问nginx

curl 10.109.145.67

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...

在集群外访问nginx

curl 172.16.64.233:32538
curl 172.16.64.232:32538
curl 172.16.64.235:32538

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...

至此,一个k8s集群在生产环境的模拟安装,就结束了。

最后修改:2021 年 06 月 09 日 10 : 33 PM
如果觉得我的文章对你有用,请随意赞赏