Kubernetes kubeadm离线部署

博主：田晓勇
发布时间：2021 年 06 月 09 日
350次浏览
暂无评论
5870字数
分类： Kubernetes
百度已收录

为什么使用kubeadm安装Kubenetes集群？

kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。这个工具能通过两条指令快速完成一个kubernetes集群的部署。
网上很多人说通过二进制安装能了解到配置的细节，其实通过kubeadm安装也能查看到配置的细节。可以自动生成证书，对初学者带来了不少便利。

网络环境

我们完全模拟生产环境中，不可以访问外部互联网的情况。
基础的yum源是有提供的，像什么docker-ce、kubernetes的源是没有的。
k8s.gcr.io、quay.io这些域名也是不可以访问的。

环境准备

主机名称	节点类型	IP地址
k8s-master	master节点	172.16.64.233
k8s-node1	node节点	172.16.64.232
k8s-node2	node节点	172.16.64.235

master节点需要至少2个CPU，不然会报如错误：

error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

关闭selinux

把SELINUX=enforcing替换成SELINUX=disabled

sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config
setenforce 0

查看一下selinux的状态。

getenforce
Permissive

关闭Swap

swapoff -a
cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak | grep -v swap > /etc/fstab

grep -v swap是查找不包含swap的行。
查看一下swap的情况，Swap已经全部为0了。

free -m
              total        used        free      shared  buff/cache   available
Mem:            972         142         715           7         114         699
Swap:             0           0           0

设置主机名

在master节点上设置主机名。

hostnamectl set-hostname k8s-master

在node1节点上设置主机名。

hostnamectl set-hostname k8s-node1

在node2节点上设置主机名。

hostnamectl set-hostname k8s-node2

在master上查看主机名。

hostname
k8s-master

设置hosts

cat >> /etc/hosts <<EOF
172.16.64.233   k8s-master
172.16.64.232   k8s-node1
172.16.64.235   k8s-node2
EOF

修改sysctl.conf

暂时未修改，装docker的时候会自动修改。可以暂时先跳过这一步。
如果未修改成功，在执行docker info命令时，会显示如下提示信息。

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

cat /proc/sys/net/bridge/bridge-nf-call-iptables
0
cat /proc/sys/net/bridge/bridge-nf-call-ip6tables
0

可通过以下方法来做修改。

# 修改 /etc/sysctl.conf
# 如果有配置，则修改
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g"  /etc/sysctl.conf
# 可能没有，追加
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
sysctl -p

也就是在/etc/sysctl.conf末尾加上如下内容：

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

同时让配置生效sysctl -p

安装Docker

下载Docker

由于我们在生产环境中是没法连接互联网的，所以要提前准备好docker rpm包。
我们在另一台可以联网的机器上下载安装所需的软件。

添加 Docker yum源

在联网的机器上，下载docker，配置docker-ce源。

cd /etc/yum.repos.d/
wget https://download.docker.com/linux/centos/docker-ce.repo

查看Docker版本

yum list docker-ce --showduplicates
...
docker-ce.x86_64                            18.06.3.ce-3.el7                                   docker-ce-stable
docker-ce.x86_64                            3:18.09.0-3.el7                                    docker-ce-stable
...

我们选择安装docker-ce.18.06.3.ce-3.el7

执行下载Docker

yum install --downloadonly --downloaddir ~/k8s/docker docker-ce-18.06.3.ce-3.el7

docker及其依赖会下载到~/docker文件夹中。

我们可以看到只有docker-ce是来自docker-ce-stable源的。

===============================================================================================================
 Package                          架构             版本                       源                          大小
===============================================================================================================
正在安装:
 docker-ce                        x86_64           18.06.3.ce-3.el7           docker-ce-stable            41 M
为依赖而安装:
 audit-libs-python                x86_64           2.8.5-4.el7                base                        76 k
 checkpolicy                      x86_64           2.5-8.el7                  base                       295 k
 container-selinux                noarch           2:2.107-3.el7              extras                      39 k
 libcgroup                        x86_64           0.41-21.el7                base                        66 k
 libseccomp                       x86_64           2.3.1-3.el7                base                        56 k
 libsemanage-python               x86_64           2.5-14.el7                 base                       113 k
 libtool-ltdl                     x86_64           2.4.2-22.el7_3             base                        49 k
 policycoreutils-python           x86_64           2.5-33.el7                 base                       457 k
 python-IPy                       noarch           0.75-6.el7                 base                        32 k
 setools-libs                     x86_64           3.3.8-4.el7                base                       620 k

所以，我们只需要把docker-ce-18.06.3.ce-3.el7.x86_64.rpm拷贝到master及node节点里面。
在master及node节点里创建~/k8s/docker目录，用于存放docker安装rpm包。

mkdir -p ~/k8s/docker

拷贝到k8s集群

scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.233:~/k8s/docker/
scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.232:~/k8s/docker/
scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.235:~/k8s/docker/

安装Docker

yum install k8s/docker/docker-ce-18.06.3.ce-3.el7.x86_64.rpm

设置开机启动

systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

我们可以查看一下安装包到底生成了哪些文件。

rpm -ql docker-ce

或者

rpm -qpl k8s/docker/docker-ce-18.06.3.ce-3.el7.x86_64.rpm

启动Docker

systemctl start docker

查看Docker信息

docker info
...
Cgroup Driver: cgroupfs
...

呆会儿我们还需要修改这个值。

安装k8s组件

由于kubeadm是依赖kubelet, kubectl的，所以我们只需要下载kubeadm的rpm，其依赖就自动下载下来了。但是版本可能不是我们想要的，所以可能需要单独下载。比如我下载kubeadm-1.15.6，它依赖的可能是kubelet-1.16.x。

下载k8s组件

我们需要安装kubeadm, kubelet, kubectl，版本需要一致。在可以连外网的机器上下载组件，同上面docker。

添加kubernetes yum源

cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enabled=1
EOF

查看kubeadm版本

yum list kubeadm --showduplicates

kubeadm.x86_64                                                         1.15.6-0

下载kubeadm-1.15.6

yum install --downloadonly --downloaddir ~/k8s/kubernetes kubeadm-1.15.6

根据如下依赖关系

====================================================================================================================================================
 Package                                    架构                       版本                                    源                              大小
====================================================================================================================================================
正在安装:
 kubeadm                                    x86_64                     1.15.6-0                                kubernetes                     8.9 M
为依赖而安装:
 conntrack-tools                            x86_64                     1.4.4-5.el7_7.2                         updates                        187 k
 cri-tools                                  x86_64                     1.13.0-0                                kubernetes                     5.1 M
 kubectl                                    x86_64                     1.16.3-0                                kubernetes                      10 M
 kubelet                                    x86_64                     1.16.3-0                                kubernetes                      22 M
 kubernetes-cni                             x86_64                     0.7.5-0                                 kubernetes                      10 M
 libnetfilter_cthelper                      x86_64                     1.0.0-10.el7_7.1                        updates                         18 k
 libnetfilter_cttimeout                     x86_64                     1.0.0-6.el7_7.1                         updates                         18 k
 libnetfilter_queue                         x86_64                     1.0.2-2.el7_2                           base                            23 k
 socat                                      x86_64                     1.7.3.2-2.el7                           base

我们只需要把来自kubernetes源的kubeadm和4个依赖cri-tools, kubectl, kubelet和kubernetes-cni拷贝到master和node节点。

下载kubelet-1.15.6

yum install --downloadonly --downloaddir ~/k8s/kubernetes kubelet-1.15.6

下载kubectl-1.15.6

yum install --downloadonly --downloaddir ~/k8s/kubernetes kubectl-1.15.6

拷贝到k8s集群

在master及node节点里创建~/k8s/kubernetes目录，用于存放k8s组件安装的rpm包。

mkdir -p ~/k8s/kubernetes

kubeadm

cri-tools

scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubectl

scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubelet

scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubernetes-cni

scp 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

安装k8s组件

yum install ~/k8s/kubernetes/*.rpm

这样，kubeadm, kubectl, kubelet就已经安装好了。
设置kubelet的开机启动。我们并不需要启动kubelet，就算启动，也是不能成功的。执行kubeadm命令，会生成一些配置文件，这时才会让kubelet启动成功的。

systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

拉取镜像

执行kubeadm时，需要用到一些镜像，我们需要提前准备。

查看需要依赖哪些镜像

kubeadm config images list
W1207 18:53:23.129020   10255 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1207 18:53:23.129433   10255 version.go:99] falling back to the local client version: v1.15.6
k8s.gcr.io/kube-apiserver:v1.15.6
k8s.gcr.io/kube-controller-manager:v1.15.6
k8s.gcr.io/kube-scheduler:v1.15.6
k8s.gcr.io/kube-proxy:v1.15.6
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

在生产环境，是肯定访问不了k8s.gcr.io这个地址的。在有大陆联网的机器上，也是无法访问的。所以我们需要使用国内镜像先下载下来。

镜像地址，请参考 K8S国内的镜像源

拉取镜像（在三台机器上拉取如下镜像）

docker pull gcr.azk8s.cn/google-containers/kube-apiserver:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-controller-manager:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-scheduler:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-proxy:v1.15.6
docker pull gcr.azk8s.cn/google-containers/pause:3.1
docker pull gcr.azk8s.cn/google-containers/etcd:3.3.10
docker pull gcr.azk8s.cn/google-containers/coredns:1.3.1

查看拉取镜像。

docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
gcr.azk8s.cn/google-containers/kube-proxy                v1.15.6             d756327a2327        3 weeks ago         82.4MB
gcr.azk8s.cn/google-containers/kube-apiserver            v1.15.6             9f612b9e9bbf        3 weeks ago         207MB
gcr.azk8s.cn/google-containers/kube-controller-manager   v1.15.6             83ab61bd43ad        3 weeks ago         159MB
gcr.azk8s.cn/google-containers/kube-scheduler            v1.15.6             502e54938456        3 weeks ago         81.1MB
gcr.azk8s.cn/google-containers/coredns                   1.3.1               eb516548c180        10 months ago       40.3MB
gcr.azk8s.cn/google-containers/etcd                      3.3.10              2c4adeb21b4f        12 months ago       258MB
gcr.azk8s.cn/google-containers/pause                     3.1                 da86e6ba6ca1        23 months ago       742kB

tag镜像

为了让kubeadm程序能找到k8s.gcr.io下面的镜像，需要把刚才下载的镜像名称重新打一下tag。

docker images | grep gcr.azk8s.cn/google-containers | sed 's/gcr.azk8s.cn\/google-containers/k8s.gcr.io/' | awk '{print "docker tag " $3 " " $1 ":" $2}' | sh

删除旧的镜像，当然，你留着也不会占用太多空间。

docker images | grep gcr.azk8s.cn/google-containers | awk '{print "docker rmi " $1 ":" $2}' | sh

查看镜像

docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                v1.15.6             d756327a2327        3 weeks ago         82.4MB
k8s.gcr.io/kube-apiserver            v1.15.6             9f612b9e9bbf        3 weeks ago         207MB
k8s.gcr.io/kube-controller-manager   v1.15.6             83ab61bd43ad        3 weeks ago         159MB
k8s.gcr.io/kube-scheduler            v1.15.6             502e54938456        3 weeks ago         81.1MB
k8s.gcr.io/coredns                   1.3.1               eb516548c180        10 months ago       40.3MB
k8s.gcr.io/etcd                      3.3.10              2c4adeb21b4f        12 months ago       258MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        23 months ago       742kB

镜像搞定了。

部署k8s集群

初始化master节点

在master节点上执行kubeadm init命令。
我们接下来首先会使用flannel网络，所以参数中必须设置--pod-network-cidr=10.244.0.0/16，这个IP地址是固定的。

kubeadm init \
--apiserver-advertise-address=172.16.64.233 \
--pod-network-cidr=10.244.0.0/16

W1207 21:18:48.257967   10859 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1207 21:18:48.258448   10859 version.go:99] falling back to the local client version: v1.15.6
[init] Using Kubernetes version: v1.15.6
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.64.233]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [172.16.64.233 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [172.16.64.233 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 36.504799 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 1czxp7.4tt0x3lxdcus8wer
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.64.233:6443 --token 1czxp7.4tt0x3lxdcus8wer \
    --discovery-token-ca-cert-hash sha256:abb676401e56a48f07675ff802f1abedd512cce0523190b2e0f636ee6d70d8b4

解决WARNING

我们看到上面的消息中有一句

[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

还记得前面我在查看docker info时，有提到要修改cgroup driver么？现在就来修改吧。

修改或创建/etc/docker/daemon.json，添加如下内容：

{
    "exec-opts": ["native.cgroupdriver=systemd"]
}

重启docker

systemctl restart docker

查看修改结果，如果Cgroup Driver改为systemd后就表示成功了。

docker info
...
Cgroup Driver: systemd
...

重置环境

kubeadm reset

[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W1207 22:12:18.285935   27649 reset.go:98] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get https://172.16.64.233:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: dial tcp 172.16.64.233:6443: connect: connection refused
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W1207 22:12:19.569005   27649 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

再次初始化Master节点

apiserver-advertise-address和pod-network-cidr参数都可以省略掉。

kubeadm init \
--apiserver-advertise-address=172.16.64.233 \
--pod-network-cidr=10.244.0.0/16

...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.64.233:6443 --token duof19.l9q3dsh4ccen4ya0 \
    --discovery-token-ca-cert-hash sha256:66fad0ed5f46f5ea9a394276a77db16d26d60465ec9930f8c21aa924a5df9bb5

提示信息和上面初始化时的信息一样，只是少了刚才的WARNING。

按照信息提示，执行如下命令，目前登录的就是root用户，所以也不需要用sudo了。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看节点信息，节点状态为NotReady:

kubectl get no
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   2m22s   v1.15.6

往集群里面加入node节点

在节点node1上，按上面的提示执行命令：

kubeadm join 172.16.64.233:6443 --token duof19.l9q3dsh4ccen4ya0 \
    --discovery-token-ca-cert-hash sha256:66fad0ed5f46f5ea9a394276a77db16d26d60465ec9930f8c21aa924a5df9bb5
    
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

在Master节点上（control-plane)上查看节点信息

kubectl get no
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   NotReady   master   7m    v1.15.6
k8s-node1    NotReady   <none>   65s   v1.15.6

我们看到了多了一个节点，虽然现在都是NotReady状态。

Token过期后再加入节点

过了一段时间后，再加入节点，这个时候会提示token已经过期了。我们可以这样拿到token和hash值。

kubeadm token create
kubeadm token list
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

安装Network插件

查看安装方法

查看flannel的官网https://github.com/coreos/flannel，找到安装方法。

For Kubernetes v1.7+ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

下载yml文件

在有网络的机器上下载kube-flannel.yml文件

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

把下载好的yml文件分发到k8s集群的三台机器里面。

下载镜像

cat kube-flannel.yml | grep image
        image: quay.io/coreos/flannel:v0.11.0-amd64
        ...

还记得前面方法么？不记得就回到上面再看看吧。

docker pull quay.azk8s.cn/coreos/flannel:v0.11.0-amd64
docker tag ff281650a721 quay.io/coreos/flannel:v0.11.0-amd64
docker rmi quay.azk8s.cn/coreos/flannel:v0.11.0-amd64

安装flannel

我们也可以选择安装Calico网络插件。
在Master节点执行：

kubectl apply -f kube-flannel.yml

podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

查看节点信息

kubectl get no
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   33m   v1.15.6
k8s-node1    Ready    <none>   28m   v1.15.6
k8s-node2    Ready    <none>   17m   v1.15.6

这一下所有节点都已经ready了。

Master节点

ps -ef | grep kube
root       1652      1  3 15:13 ?        00:00:04 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1
root       1973   1909  8 15:13 ?        00:00:12 kube-apiserver --advertise-address=172.16.64.233 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root       1980   1915  1 15:13 ?        00:00:01 kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
root       1995   1936  2 15:13 ?        00:00:03 etcd --advertise-client-urls=https://172.16.64.233:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://172.16.64.233:2380 --initial-cluster=k8s-master=https://172.16.64.233:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://172.16.64.233:2379 --listen-peer-urls=https://172.16.64.233:2380 --name=k8s-master --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
root       2002   1951  2 15:13 ?        00:00:03 kube-controller-manager --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true
root       2263   2244  0 15:13 ?        00:00:00 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8s-master
root       3907   3888  0 15:14 ?        00:00:00 /usr/bin/kube-controllers

Worker节点

ps -ef | grep kube
root       1355      1  1 15:05 ?        00:00:06 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1
root       1669   1620  0 15:13 ?        00:00:00 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8s-node1

测试k8s集群

安装一个nginx。

创建一个部署(deployment)

在master节点（Control Plane）安装一个叫nginx-deployment的deployment：

kubectl create deploy nginx-deployment --image=nginx
deployment.apps/nginx-deployment created

查看deployment状态

kubectl get deploy
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   0/1     1            0           44s

发现没有READY。

查看pod状态

kubectl get po
NAME                                READY   STATUS              RESTARTS   AGE
nginx-deployment-6f77f65499-htdps   0/1     ContainerCreating   0          111s

也是没有READY。继续查看详细信息。

kubectl describe po nginx-deployment-6f77f65499-htdps

...
Events:
  Type    Reason     Age    From                Message
  ----    ------     ----   ----                -------
  Normal  Scheduled  2m18s  default-scheduler   Successfully assigned default/nginx-deployment-6f77f65499-htdps to k8s-node1
  Normal  Pulling    2m17s  kubelet, k8s-node1  Pulling image "nginx"

初步判断应该是拉取镜像拉不下来，或者速度非常慢。

配置docker源

在生产环境，肯定是有内部的镜像源的，在这里，我就模拟把源配置为阿里的镜像源了。
/etc/docker/daemon.json内容如下：

{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "registry-mirrors": ["http://hub-mirror.c.163.com"]
}

重启docker

systemctl restart docker

这个时候，镜像就容易拉取了。

测试pod

kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-6f77f65499-htdps   1/1     Running   0          26m   10.244.1.3   k8s-node1   <none>           <none>

我们看到pod的IP为10.244.1.3。还记得我们在初始化master节点时设置的参数--pod-network-cidr=10.244.0.0/16么？

在集群内的三个节点访问nginx，能成功访问。

curl 10.244.1.3

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...

创建Service

我们把deployment暴露出来。

kubectl expose deploy nginx-deployment --port=80 --type=NodePort
service/nginx-deployment exposed

查看状态

kubectl get svc
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes         ClusterIP   10.96.0.1       <none>        443/TCP        71m
nginx-deployment   NodePort    10.109.145.67   <none>        80:32538/TCP   32s

在三个节点内访问nginx

curl 10.109.145.67

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...

在集群外访问nginx

curl 172.16.64.233:32538
curl 172.16.64.232:32538
curl 172.16.64.235:32538

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...

至此，一个k8s集群在生产环境的模拟安装，就结束了。

最后修改：2021 年 06 月 09 日 10 : 33 PM

如果觉得我的文章对你有用，请随意赞赏

发表评论取消回复

评论 *

私密评论

名称 *

邮箱 *

地址

乌拉拉
OωO好棒呀！！！
fank
ヾ(≧∇≦*)ゝ云服务器代理，有需要可以联系，qq 1...
你哥
你这么吊，我同意了吗？
吴承恩
啊啊啊

Kubernetes kubeadm离线部署

田晓勇 • 2021 年 06 月 09 日

为什么使用kubeadm安装Kubenetes集群？

网络环境

环境准备

主机名称	节点类型	IP地址
k8s-master	master节点	172.16.64.233
k8s-node1	node节点	172.16.64.232
k8s-node2	node节点	172.16.64.235

master节点需要至少2个CPU，不然会报如错误：

error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

关闭selinux

把SELINUX=enforcing替换成SELINUX=disabled

sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config
setenforce 0

查看一下selinux的状态。

getenforce
Permissive

关闭Swap

swapoff -a
cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak | grep -v swap > /etc/fstab

grep -v swap是查找不包含swap的行。
查看一下swap的情况，Swap已经全部为0了。

free -m
              total        used        free      shared  buff/cache   available
Mem:            972         142         715           7         114         699
Swap:             0           0           0

设置主机名

在master节点上设置主机名。

hostnamectl set-hostname k8s-master

在node1节点上设置主机名。

hostnamectl set-hostname k8s-node1

在node2节点上设置主机名。

hostnamectl set-hostname k8s-node2

在master上查看主机名。

hostname
k8s-master

设置hosts

cat >> /etc/hosts <<EOF
172.16.64.233   k8s-master
172.16.64.232   k8s-node1
172.16.64.235   k8s-node2
EOF

修改sysctl.conf

暂时未修改，装docker的时候会自动修改。可以暂时先跳过这一步。
如果未修改成功，在执行docker info命令时，会显示如下提示信息。

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

cat /proc/sys/net/bridge/bridge-nf-call-iptables
0
cat /proc/sys/net/bridge/bridge-nf-call-ip6tables
0

可通过以下方法来做修改。

# 修改 /etc/sysctl.conf
# 如果有配置，则修改
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g"  /etc/sysctl.conf
# 可能没有，追加
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
sysctl -p

也就是在/etc/sysctl.conf末尾加上如下内容：

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

同时让配置生效sysctl -p

安装Docker

下载Docker

由于我们在生产环境中是没法连接互联网的，所以要提前准备好docker rpm包。
我们在另一台可以联网的机器上下载安装所需的软件。

添加 Docker yum源

在联网的机器上，下载docker，配置docker-ce源。

cd /etc/yum.repos.d/
wget https://download.docker.com/linux/centos/docker-ce.repo

查看Docker版本

yum list docker-ce --showduplicates
...
docker-ce.x86_64                            18.06.3.ce-3.el7                                   docker-ce-stable
docker-ce.x86_64                            3:18.09.0-3.el7                                    docker-ce-stable
...

我们选择安装docker-ce.18.06.3.ce-3.el7

执行下载Docker

yum install --downloadonly --downloaddir ~/k8s/docker docker-ce-18.06.3.ce-3.el7

docker及其依赖会下载到~/docker文件夹中。

我们可以看到只有docker-ce是来自docker-ce-stable源的。

===============================================================================================================
 Package                          架构             版本                       源                          大小
===============================================================================================================
正在安装:
 docker-ce                        x86_64           18.06.3.ce-3.el7           docker-ce-stable            41 M
为依赖而安装:
 audit-libs-python                x86_64           2.8.5-4.el7                base                        76 k
 checkpolicy                      x86_64           2.5-8.el7                  base                       295 k
 container-selinux                noarch           2:2.107-3.el7              extras                      39 k
 libcgroup                        x86_64           0.41-21.el7                base                        66 k
 libseccomp                       x86_64           2.3.1-3.el7                base                        56 k
 libsemanage-python               x86_64           2.5-14.el7                 base                       113 k
 libtool-ltdl                     x86_64           2.4.2-22.el7_3             base                        49 k
 policycoreutils-python           x86_64           2.5-33.el7                 base                       457 k
 python-IPy                       noarch           0.75-6.el7                 base                        32 k
 setools-libs                     x86_64           3.3.8-4.el7                base                       620 k

所以，我们只需要把docker-ce-18.06.3.ce-3.el7.x86_64.rpm拷贝到master及node节点里面。
在master及node节点里创建~/k8s/docker目录，用于存放docker安装rpm包。

mkdir -p ~/k8s/docker

拷贝到k8s集群

scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.233:~/k8s/docker/
scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.232:~/k8s/docker/
scp docker-ce-18.06.3.ce-3.el7.x86_64.rpm root@172.16.64.235:~/k8s/docker/

安装Docker

yum install k8s/docker/docker-ce-18.06.3.ce-3.el7.x86_64.rpm

设置开机启动

systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

我们可以查看一下安装包到底生成了哪些文件。

rpm -ql docker-ce

或者

rpm -qpl k8s/docker/docker-ce-18.06.3.ce-3.el7.x86_64.rpm

启动Docker

systemctl start docker

查看Docker信息

docker info
...
Cgroup Driver: cgroupfs
...

呆会儿我们还需要修改这个值。

安装k8s组件

下载k8s组件

我们需要安装kubeadm, kubelet, kubectl，版本需要一致。在可以连外网的机器上下载组件，同上面docker。

添加kubernetes yum源

cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enabled=1
EOF

查看kubeadm版本

yum list kubeadm --showduplicates

kubeadm.x86_64                                                         1.15.6-0

下载kubeadm-1.15.6

yum install --downloadonly --downloaddir ~/k8s/kubernetes kubeadm-1.15.6

根据如下依赖关系

====================================================================================================================================================
 Package                                    架构                       版本                                    源                              大小
====================================================================================================================================================
正在安装:
 kubeadm                                    x86_64                     1.15.6-0                                kubernetes                     8.9 M
为依赖而安装:
 conntrack-tools                            x86_64                     1.4.4-5.el7_7.2                         updates                        187 k
 cri-tools                                  x86_64                     1.13.0-0                                kubernetes                     5.1 M
 kubectl                                    x86_64                     1.16.3-0                                kubernetes                      10 M
 kubelet                                    x86_64                     1.16.3-0                                kubernetes                      22 M
 kubernetes-cni                             x86_64                     0.7.5-0                                 kubernetes                      10 M
 libnetfilter_cthelper                      x86_64                     1.0.0-10.el7_7.1                        updates                         18 k
 libnetfilter_cttimeout                     x86_64                     1.0.0-6.el7_7.1                         updates                         18 k
 libnetfilter_queue                         x86_64                     1.0.2-2.el7_2                           base                            23 k
 socat                                      x86_64                     1.7.3.2-2.el7                           base

我们只需要把来自kubernetes源的kubeadm和4个依赖cri-tools, kubectl, kubelet和kubernetes-cni拷贝到master和node节点。

下载kubelet-1.15.6

yum install --downloadonly --downloaddir ~/k8s/kubernetes kubelet-1.15.6

下载kubectl-1.15.6

yum install --downloadonly --downloaddir ~/k8s/kubernetes kubectl-1.15.6

拷贝到k8s集群

在master及node节点里创建~/k8s/kubernetes目录，用于存放k8s组件安装的rpm包。

mkdir -p ~/k8s/kubernetes

kubeadm

cri-tools

scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubectl

scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp 5181c2b7eee876b8ce205f0eca87db2b3d00ffd46d541882620cb05b738d7a80-kubectl-1.15.6-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubelet

scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.233:~/k8s/kubernetes/
scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.232:~/k8s/kubernetes/
scp e9e7cc53edd19d0ceb654d1bde95ec79f89d26de91d33af425ffe8464582b36e-kubelet-1.15.6-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

kubernetes-cni

scp 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm root@172.16.64.235:~/k8s/kubernetes/

安装k8s组件

yum install ~/k8s/kubernetes/*.rpm

systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

拉取镜像

执行kubeadm时，需要用到一些镜像，我们需要提前准备。

查看需要依赖哪些镜像

kubeadm config images list
W1207 18:53:23.129020   10255 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1207 18:53:23.129433   10255 version.go:99] falling back to the local client version: v1.15.6
k8s.gcr.io/kube-apiserver:v1.15.6
k8s.gcr.io/kube-controller-manager:v1.15.6
k8s.gcr.io/kube-scheduler:v1.15.6
k8s.gcr.io/kube-proxy:v1.15.6
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

在生产环境，是肯定访问不了k8s.gcr.io这个地址的。在有大陆联网的机器上，也是无法访问的。所以我们需要使用国内镜像先下载下来。

镜像地址，请参考 K8S国内的镜像源

拉取镜像（在三台机器上拉取如下镜像）

docker pull gcr.azk8s.cn/google-containers/kube-apiserver:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-controller-manager:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-scheduler:v1.15.6
docker pull gcr.azk8s.cn/google-containers/kube-proxy:v1.15.6
docker pull gcr.azk8s.cn/google-containers/pause:3.1
docker pull gcr.azk8s.cn/google-containers/etcd:3.3.10
docker pull gcr.azk8s.cn/google-containers/coredns:1.3.1

查看拉取镜像。

docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
gcr.azk8s.cn/google-containers/kube-proxy                v1.15.6             d756327a2327        3 weeks ago         82.4MB
gcr.azk8s.cn/google-containers/kube-apiserver            v1.15.6             9f612b9e9bbf        3 weeks ago         207MB
gcr.azk8s.cn/google-containers/kube-controller-manager   v1.15.6             83ab61bd43ad        3 weeks ago         159MB
gcr.azk8s.cn/google-containers/kube-scheduler            v1.15.6             502e54938456        3 weeks ago         81.1MB
gcr.azk8s.cn/google-containers/coredns                   1.3.1               eb516548c180        10 months ago       40.3MB
gcr.azk8s.cn/google-containers/etcd                      3.3.10              2c4adeb21b4f        12 months ago       258MB
gcr.azk8s.cn/google-containers/pause                     3.1                 da86e6ba6ca1        23 months ago       742kB

tag镜像

为了让kubeadm程序能找到k8s.gcr.io下面的镜像，需要把刚才下载的镜像名称重新打一下tag。

docker images | grep gcr.azk8s.cn/google-containers | sed 's/gcr.azk8s.cn\/google-containers/k8s.gcr.io/' | awk '{print "docker tag " $3 " " $1 ":" $2}' | sh

删除旧的镜像，当然，你留着也不会占用太多空间。

docker images | grep gcr.azk8s.cn/google-containers | awk '{print "docker rmi " $1 ":" $2}' | sh

查看镜像

docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                v1.15.6             d756327a2327        3 weeks ago         82.4MB
k8s.gcr.io/kube-apiserver            v1.15.6             9f612b9e9bbf        3 weeks ago         207MB
k8s.gcr.io/kube-controller-manager   v1.15.6             83ab61bd43ad        3 weeks ago         159MB
k8s.gcr.io/kube-scheduler            v1.15.6             502e54938456        3 weeks ago         81.1MB
k8s.gcr.io/coredns                   1.3.1               eb516548c180        10 months ago       40.3MB
k8s.gcr.io/etcd                      3.3.10              2c4adeb21b4f        12 months ago       258MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        23 months ago       742kB

镜像搞定了。

部署k8s集群

初始化master节点

在master节点上执行kubeadm init命令。
我们接下来首先会使用flannel网络，所以参数中必须设置--pod-network-cidr=10.244.0.0/16，这个IP地址是固定的。

kubeadm init \
--apiserver-advertise-address=172.16.64.233 \
--pod-network-cidr=10.244.0.0/16

W1207 21:18:48.257967   10859 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1207 21:18:48.258448   10859 version.go:99] falling back to the local client version: v1.15.6
[init] Using Kubernetes version: v1.15.6
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.64.233]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [172.16.64.233 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [172.16.64.233 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 36.504799 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 1czxp7.4tt0x3lxdcus8wer
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.64.233:6443 --token 1czxp7.4tt0x3lxdcus8wer \
    --discovery-token-ca-cert-hash sha256:abb676401e56a48f07675ff802f1abedd512cce0523190b2e0f636ee6d70d8b4

解决WARNING

我们看到上面的消息中有一句

[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

还记得前面我在查看docker info时，有提到要修改cgroup driver么？现在就来修改吧。

修改或创建/etc/docker/daemon.json，添加如下内容：

{
    "exec-opts": ["native.cgroupdriver=systemd"]
}

重启docker

systemctl restart docker

查看修改结果，如果Cgroup Driver改为systemd后就表示成功了。

docker info
...
Cgroup Driver: systemd
...

重置环境

kubeadm reset

[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W1207 22:12:18.285935   27649 reset.go:98] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get https://172.16.64.233:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: dial tcp 172.16.64.233:6443: connect: connection refused
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W1207 22:12:19.569005   27649 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

再次初始化Master节点

apiserver-advertise-address和pod-network-cidr参数都可以省略掉。

kubeadm init \
--apiserver-advertise-address=172.16.64.233 \
--pod-network-cidr=10.244.0.0/16

...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.64.233:6443 --token duof19.l9q3dsh4ccen4ya0 \
    --discovery-token-ca-cert-hash sha256:66fad0ed5f46f5ea9a394276a77db16d26d60465ec9930f8c21aa924a5df9bb5

提示信息和上面初始化时的信息一样，只是少了刚才的WARNING。

按照信息提示，执行如下命令，目前登录的就是root用户，所以也不需要用sudo了。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看节点信息，节点状态为NotReady:

kubectl get no
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   2m22s   v1.15.6

往集群里面加入node节点

在节点node1上，按上面的提示执行命令：

kubeadm join 172.16.64.233:6443 --token duof19.l9q3dsh4ccen4ya0 \
    --discovery-token-ca-cert-hash sha256:66fad0ed5f46f5ea9a394276a77db16d26d60465ec9930f8c21aa924a5df9bb5
    
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

在Master节点上（control-plane)上查看节点信息

kubectl get no
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   NotReady   master   7m    v1.15.6
k8s-node1    NotReady   <none>   65s   v1.15.6

我们看到了多了一个节点，虽然现在都是NotReady状态。

Token过期后再加入节点

过了一段时间后，再加入节点，这个时候会提示token已经过期了。我们可以这样拿到token和hash值。

kubeadm token create
kubeadm token list
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

安装Network插件

查看安装方法

查看flannel的官网https://github.com/coreos/flannel，找到安装方法。

For Kubernetes v1.7+ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

下载yml文件

在有网络的机器上下载kube-flannel.yml文件

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

把下载好的yml文件分发到k8s集群的三台机器里面。

下载镜像

cat kube-flannel.yml | grep image
        image: quay.io/coreos/flannel:v0.11.0-amd64
        ...

还记得前面方法么？不记得就回到上面再看看吧。

docker pull quay.azk8s.cn/coreos/flannel:v0.11.0-amd64
docker tag ff281650a721 quay.io/coreos/flannel:v0.11.0-amd64
docker rmi quay.azk8s.cn/coreos/flannel:v0.11.0-amd64

安装flannel

我们也可以选择安装Calico网络插件。
在Master节点执行：

kubectl apply -f kube-flannel.yml

podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

查看节点信息

kubectl get no
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   33m   v1.15.6
k8s-node1    Ready    <none>   28m   v1.15.6
k8s-node2    Ready    <none>   17m   v1.15.6

这一下所有节点都已经ready了。

Master节点

ps -ef | grep kube
root       1652      1  3 15:13 ?        00:00:04 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1
root       1973   1909  8 15:13 ?        00:00:12 kube-apiserver --advertise-address=172.16.64.233 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root       1980   1915  1 15:13 ?        00:00:01 kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
root       1995   1936  2 15:13 ?        00:00:03 etcd --advertise-client-urls=https://172.16.64.233:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://172.16.64.233:2380 --initial-cluster=k8s-master=https://172.16.64.233:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://172.16.64.233:2379 --listen-peer-urls=https://172.16.64.233:2380 --name=k8s-master --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
root       2002   1951  2 15:13 ?        00:00:03 kube-controller-manager --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true
root       2263   2244  0 15:13 ?        00:00:00 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8s-master
root       3907   3888  0 15:14 ?        00:00:00 /usr/bin/kube-controllers

Worker节点

ps -ef | grep kube
root       1355      1  1 15:05 ?        00:00:06 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1
root       1669   1620  0 15:13 ?        00:00:00 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=k8s-node1

测试k8s集群

安装一个nginx。

创建一个部署(deployment)

在master节点（Control Plane）安装一个叫nginx-deployment的deployment：

kubectl create deploy nginx-deployment --image=nginx
deployment.apps/nginx-deployment created

查看deployment状态

kubectl get deploy
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   0/1     1            0           44s

发现没有READY。

查看pod状态

kubectl get po
NAME                                READY   STATUS              RESTARTS   AGE
nginx-deployment-6f77f65499-htdps   0/1     ContainerCreating   0          111s

也是没有READY。继续查看详细信息。

kubectl describe po nginx-deployment-6f77f65499-htdps

...
Events:
  Type    Reason     Age    From                Message
  ----    ------     ----   ----                -------
  Normal  Scheduled  2m18s  default-scheduler   Successfully assigned default/nginx-deployment-6f77f65499-htdps to k8s-node1
  Normal  Pulling    2m17s  kubelet, k8s-node1  Pulling image "nginx"

初步判断应该是拉取镜像拉不下来，或者速度非常慢。

配置docker源

在生产环境，肯定是有内部的镜像源的，在这里，我就模拟把源配置为阿里的镜像源了。
/etc/docker/daemon.json内容如下：

{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "registry-mirrors": ["http://hub-mirror.c.163.com"]
}

重启docker

systemctl restart docker

这个时候，镜像就容易拉取了。

测试pod

kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
nginx-deployment-6f77f65499-htdps   1/1     Running   0          26m   10.244.1.3   k8s-node1   <none>           <none>

我们看到pod的IP为10.244.1.3。还记得我们在初始化master节点时设置的参数--pod-network-cidr=10.244.0.0/16么？

在集群内的三个节点访问nginx，能成功访问。

curl 10.244.1.3

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...

创建Service

我们把deployment暴露出来。

kubectl expose deploy nginx-deployment --port=80 --type=NodePort
service/nginx-deployment exposed

查看状态

kubectl get svc
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes         ClusterIP   10.96.0.1       <none>        443/TCP        71m
nginx-deployment   NodePort    10.109.145.67   <none>        80:32538/TCP   32s

在三个节点内访问nginx

curl 10.109.145.67

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...

在集群外访问nginx

curl 172.16.64.233:32538
curl 172.16.64.232:32538
curl 172.16.64.235:32538

...
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
...

至此，一个k8s集群在生产环境的模拟安装，就结束了。

为什么使用kubeadm安装Kubenetes集群？

网络环境

环境准备

关闭防火墙

关闭selinux

关闭Swap

设置主机名

设置hosts

修改sysctl.conf

安装Docker

安装k8s组件

安装k8s组件

拉取镜像

部署k8s集群

解决WARNING

往集群里面加入node节点

安装Network插件

查看节点信息

测试k8s集群

发表评论 取消回复

Kubernetes kubeadm离线部署

为什么使用kubeadm安装Kubenetes集群？

网络环境

环境准备

关闭防火墙

关闭selinux

关闭Swap

设置主机名

设置hosts

修改sysctl.conf

安装Docker

安装k8s组件

安装k8s组件

拉取镜像

部署k8s集群

解决WARNING

往集群里面加入node节点

安装Network插件

查看节点信息

测试k8s集群

发表评论取消回复