Day 34/40 : Upgrade a Multi Node Kubernetes Cluster With Kubeadm

Kubernetes is a powerful platform for managing containerized applications, but to harness its full potential, you need to keep your clusters up-to-date with the latest features and security patches. With new releases coming frequently, upgrading becomes essential for staying secure and using the latest features. This blog post will walk you through the step-by-step process of upgrading a multi-node Kubernetes cluster using kubeadm.

Prerequisites

Before beginning the upgrade process, ensure the following are in place:

• Multi-node Kubernetes Cluster: A working cluster with at least one master and multiple worker nodes.

• Kubeadm Installed on All Nodes: The cluster was initialized using kubeadm and has kubeadm, kubelet, and kubectl installed.

• Backup of Cluster Configuration: It's critical to back up your cluster, including etcd and all configuration files, before performing an upgrade.

Upgrade Steps

Upgrading a Kubernetes cluster involves several critical steps. The main process is divided between upgrading the control plane and upgrading the worker nodes. Below are the detailed steps for upgrading both parts.

1. Update Package Repositories

To ensure your nodes are pulling in the latest versions of kubeadm, kubectl, and kubelet, begin by updating the package repositories on each node.

sudo apt-get update && sudo apt-get upgrade -y

This command ensures your node is running the latest system packages, an important first step before any Kubernetes upgrade.

2. Upgrade Control Plane Components

Start by upgrading the master nodes, which host the control plane. Begin by connecting to your master node and following these steps:

• Run Preflight Check: Execute the following command to get a detailed plan of your upgrade path.

sudo kubeadm upgrade plan

This command checks the current Kubernetes version and suggests the next available version for an upgrade, along with the steps required.

• Apply the Upgrade: Once you've reviewed the upgrade plan, apply it.

sudo kubeadm upgrade apply vX.Y.Z

Here, replace vX.Y.Z with the latest available Kubernetes version. This command upgrades all control plane components (API Server, Controller Manager, Scheduler).

• Update kubelet and kubectl: After upgrading the control plane, update the kubelet and kubectl binaries on the master node.

sudo apt-get install -y kubelet kubectl
sudo systemctl restart kubelet

3. Upgrade Worker Nodes

Once the master node(s) are upgraded, it's time to upgrade the worker nodes one by one to avoid downtime:

• Drain the Node: Before upgrading a worker node, drain it so no workloads are scheduled on it during the upgrade.

kubectl drain <node-name> --ignore-daemonsets

This ensures no pods (except DaemonSets) are running on the node.

• Upgrade the Node: SSH into the worker node and upgrade it.

sudo kubeadm upgrade node
sudo apt-get install -y kubelet kubectl
sudo systemctl restart kubelet

• Uncordon the Node: Once the upgrade is complete, uncordon the node to allow workloads to be scheduled on it again.

kubectl uncordon <node-name>

Repeat these steps for each worker node in your cluster.

4. Verify the Upgrade

After upgrading all nodes (control plane and workers), you should verify that everything is functioning correctly:

• Check Node Status: Run the following command to ensure all nodes are in a "Ready" state.

kubectl get nodes

• Check Cluster Components: Verify that key components like the API server, controller manager, and scheduler are running correctly:

kubectl get pods -n kube-system

Additional Considerations

While upgrading a Kubernetes cluster is relatively straightforward, there are some additional factors to consider depending on your environment.

• Blue-Green Deployment: For a major version upgrade, you might consider implementing a blue-green deployment strategy. This involves setting up a new cluster with the latest version of Kubernetes, migrating traffic gradually, and then decommissioning the old cluster.

• Data Persistence: If your applications rely on Persistent Volumes (PVs), ensure that they are compatible with the newer version of Kubernetes. Some storage classes or CSI drivers may require updates.

• Custom Resource Definitions (CRDs): If you are using CRDs, ensure that they are updated to be compatible with the new Kubernetes API version. Some older API versions may have been deprecated and need manual changes to ensure continued functionality.

Conclusion

Upgrading a multi-node Kubernetes cluster with kubeadm may seem like a daunting task, but by following these steps, you can ensure a smooth, minimal-downtime process. Regularly upgrading your Kubernetes cluster not only helps maintain security but also enables you to take advantage of new features and improvements.

If you run into any issues or have additional questions, feel free to leave a comment below or consult the official Kubernetes upgrade guide for more detailed instructions.

Reference

https://www.youtube.com/watch?v=NtX75Ze47EU&list=PLl4APkPHzsUUOkOv3i62UidrLmSB8DcGC&index=36

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/