Kubernetes集群升级

最近公司的Kubernetes集群证书快要到期了,加上生产环境中的集群连续运行时长也已经超过一年了,很多组件伴随着官方的更新,也显得已经有一些老旧了。根据官方的要求,需要尽可能的保持运行的Kubernetes集群跟上官方的版本脚步,防止出现大的漏洞和兼容性问题,于是决定对集群进行一次升级,顺带依托升级的时候自动更新证书,也为工作证书链进行更新。

基本流程

  1. 升级主控制平面节点
  2. 升级其他控制平面节点
  3. 升级工作节点

前置条件

  1. 你需要有一个由 kubeadm 创建的 Kubernetes 集群。
  2. 禁用交换分区。
  3. 集群应使用静态的控制平面和 etcd Pod 或者 外部 etcd。
  4. 由于升级只能进行连续版本升级,务必仔细认真阅读发行说明。
  5. 务必备份所有重要组件。

升级步骤

以下内容为测试环境操作步骤记录

首先确认当前运行的kubeadm版本

[root@test01 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:56:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

同时确认kubectl版本

[root@test01 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

查看集群节点状态和版本

[root@test01 ~]# kubectl get nodes
NAME                 STATUS   ROLES    AGE   VERSION
test01.anyran   Ready    master   61d   v1.18.6
test02.anyran   Ready    master   61d   v1.18.6
test03.anyran   Ready    master   61d   v1.18.6

这里的测试环境版本在1.18.6,我们在第一个控制台平面节点查看当前的kubeadm版本信息

yum list --showduplicates kubeadm --disableexcludes=kubernetes

运行以上命令,应该会输出如下信息(节选)

Installed Packages
kubeadm.x86_64                                                                    1.18.6-0                                                                     @kubernetes
Available Packages
....................
kubeadm.x86_64                                                                    1.18.0-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.1-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.2-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.3-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.4-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.4-1                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.5-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.6-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.8-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.18.9-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.19.0-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.19.1-0                                                                     kubernetes
kubeadm.x86_64                                                                    1.19.2-0                                                                     kubernetes

可以看到当前节点运行的版本以及当前最新的版本情况,这里我们选择最新的版本作为升级测试样例
安装所选择版本的kubeadm

[root@test01 ~]# yum install kubeadm-1.19.2-0
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * epel: hk.mirrors.thegigabit.com
 * extras: mirror.bit.edu.cn
 * updates: mirror.bit.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.18.6-0 will be updated
---> Package kubeadm.x86_64 0:1.19.2-0 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================
 Package                                 Arch                                   Version                                  Repository                                  Size
==========================================================================================================================================================================
Updating:
 kubeadm                                 x86_64                                 1.19.2-0                                 kubernetes                                 8.3 M

Transaction Summary
==========================================================================================================================================================================
Upgrade  1 Package

Total download size: 8.3 M
Is this ok [y/d/N]: y

执行版本检查,确认已经更新到所需版本的kubeadm

[root@test01 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:38:53Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

版本检查无误,对当前节点的pod进行驱散并且禁用调度

[root@test01 ~]# kubectl drain test01.anyran --ignore-daemonsets
node/test01.anyran already cordoned
WARNING: ignoring DaemonSet-managed Pods: ingress-nginx/ingress-nginx-controller-g5j22, kube-system/calico-node-hn5xh, kube-system/kube-proxy-cbddp
evicting pod default/nginx-5449d4cdc5-c2nkx
evicting pod kube-system/calico-kube-controllers-578894d4cd-npb2z
evicting pod kube-system/coredns-66bff467f8-2fd9v
evicting pod kube-system/coredns-66bff467f8-kdw8r
pod/calico-kube-controllers-578894d4cd-npb2z evicted
pod/nginx-5449d4cdc5-c2nkx evicted
pod/coredns-66bff467f8-kdw8r evicted
pod/coredns-66bff467f8-2fd9v evicted
node/test01.anyran evicted

执行升级计划检查

[root@test01 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.18.6
[upgrade/versions] kubeadm version: v1.19.2
[upgrade/versions] Latest stable version: v1.19.2
[upgrade/versions] Latest stable version: v1.19.2
[upgrade/versions] Latest version in the v1.18 series: v1.18.9
[upgrade/versions] Latest version in the v1.18 series: v1.18.9

以上为节选的内容,一般说来会列出可以升级的多个版本,包含同版本内的修订版本和下一个版本,执行所需版本的升级应用

[root@test01 ~]# kubeadm upgrade apply v1.19.2
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.19.2"
[upgrade/versions] Cluster version: v1.18.6
[upgrade/versions] kubeadm version: v1.19.2
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y

升级中会自动备份配置文件,强烈建议执行之前手动对应用以及集群的重要配置进行备份,确认无误后执行升级,依据镜像拉取的快慢,所用时长也会有所不一样,直到输出以下内容表示该平面的升级完成。

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.19.2". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

手动安装对应版本的kubelet和kubectl

[root@test01 ~]# yum install kubelet-1.19.2-0 kubectl-1.19.2-0

安装完成后,对kubelet组件进行重启。

[root@test01 ~]# systemctl daemon-reload
[root@test01 ~]# systemctl restart kubelet

检查升级后的组件版本

[root@test01 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

检查升级后的集群节点状况和版本

[root@test01 ~]# kubectl get nodes
NAME                 STATUS                     ROLES    AGE   VERSION
test01.anyran   Ready,SchedulingDisabled   master   61d   v1.19.2
test02.anyran   Ready                      master   61d   v1.18.6
test03.anyran   Ready                      master   61d   v1.18.6

取消节点的隔离,允许调度

[root@test01 ~]# kubectl uncordon test01.anyran
node/test01.anyran uncordoned

升级完成后的节点状态

[root@test01 ~]# kubectl get nodes
NAME                 STATUS   ROLES    AGE   VERSION
test01.anyran   Ready    master   61d   v1.19.2
test02.anyran   Ready    master   61d   v1.18.6
test03.anyran   Ready    master   61d   v1.18.6

对第二个控制台平面节点进行升级,注意这里的命令发生了改变

[root@test02 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.19.2"...
Static pod: kube-apiserver-test02.anyran hash: 007cd47d52fb6b0cb8c8bb6004cf29ef
Static pod: kube-controller-manager-test02.anyran hash: 5f53a41ec77571f272bf23c529b24c44
Static pod: kube-scheduler-test02.anyran hash: f543c94683059cb32a4441e29fbdb238
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/etcd] Non fatal issue encountered during upgrade: the desired etcd version "3.4.13-0" is not newer than the currently installed "3.4.13-0". Skipping etcd upgrade
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests568562993"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Current and new manifests of kube-apiserver are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Current and new manifests of kube-controller-manager are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Current and new manifests of kube-scheduler are equal, skipping upgrade
[upgrade] The control plane instance for this node was successfully updated!
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

升级完成后同前一个节点一样,需要手动更新并重启kubelet组件,重启后检查集群节点状态(这里的节点隔离是手动执行的)

[root@test02 ~]# kubectl get nodes
NAME                 STATUS                     ROLES    AGE   VERSION
test01.anyran   Ready                      master   61d   v1.19.2
test02.anyran   Ready,SchedulingDisabled   master   61d   v1.19.2
test03.anyran   Ready,SchedulingDisabled   master   61d   v1.18.6

同理升级完所有的控制台平面节点,最终集群状态,可以看到集群的节点状态均为Ready,并且版本也已经更新

[root@test03 ~]# kubectl get nodes
NAME                 STATUS   ROLES    AGE   VERSION
test01.anyran   Ready    master   61d   v1.19.2
test02.anyran   Ready    master   61d   v1.19.2
test03.anyran   Ready    master   61d   v1.19.2

工作平面节点的升级方式,同后面的几个控制台节点升级方式类似。升级完成后,再检查集群的证书信息。

[root@test01 ~]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Sep 27, 2021 05:45 UTC   364d                                    no
apiserver                  Sep 27, 2021 05:45 UTC   364d            ca                      no
apiserver-etcd-client      Sep 27, 2021 05:45 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Sep 27, 2021 05:45 UTC   364d            ca                      no
controller-manager.conf    Sep 27, 2021 05:45 UTC   364d                                    no
etcd-healthcheck-client    Sep 27, 2021 05:44 UTC   364d            etcd-ca                 no
etcd-peer                  Sep 27, 2021 05:44 UTC   364d            etcd-ca                 no
etcd-server                Sep 27, 2021 05:44 UTC   364d            etcd-ca                 no
front-proxy-client         Sep 27, 2021 05:45 UTC   364d            front-proxy-ca          no
scheduler.conf             Sep 27, 2021 05:45 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jul 26, 2030 05:32 UTC   9y              no
etcd-ca                 Jul 26, 2030 05:32 UTC   9y              no
front-proxy-ca          Jul 26, 2030 05:32 UTC   9y              no

可以看到,证书也已经全部更新了(除了CA证书),又可以愉快的玩耍了。

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注