Step-by-Step Guide to Kubernetes Certificate Renewal

Let's walk through how to renew Kubernetes component certificates.

Step 1: Backing Up Important Files

Before you do anything, taking backups is always a good idea, especially when dealing with something as crucial as certificates.

# Backup your Kubernetes configuration
cp -r /etc/kubernetes/ /tmp/k8s-certs-2023/
cp -r /root/.kube/config /root/.kube/config_2023

Step 2: Check Certificate Expiration

Now, let’s check which certificates have expired:

kubeadm certs check-expiration

If you find any expired certificates, proceed with renewing them.

Step 3: Renew the Kubernetes Certificates

You can renew all of them in one go with:

kubeadm certs renew all

Once renewed, copy the admin.conf back to the .kube directory:

# Update kube config with the renewed certificates
cp /etc/kubernetes/admin.conf /root/.kube/config

Don’t forget about the Kubelet certificate:

# Regenerate kubelet certificate
rm /etc/kubernetes/kubelet.conf
kubeadm init phase kubeconfig kubelet

Restart the Kubelet service to apply the new certificates:

# Restart kubelet to apply changes
systemctl restart kubelet
systemctl status kubelet
journalctl -u kubelet -r -l | more  # Check logs if needed

You may also need to delete the existing pods in the kube-system namespace so they pick up the new configurations:

kubectl delete pod -n kube-system --all

Step 4: ETCD Certificate Renewal

What about ETCD? If you're running ETCD separately for production workloads, you'll also need to renew its certificates.

1. First, check the expiration of the ETCD client certificate:

    # Check ETCD certificate expiry
    openssl x509 -enddate -noout -in /etc/kubernetes/pki/apiserver-etcd-client.crt
   

2. Next, copy the apiserver-etcd-client.key and apiserver-etcd-client.crt from one of the ETCD nodes to the Master’s /etc/kubernetes directory. If you’re unsure, check the mount path in /etc/kubernetes/manifests/kube-apiserver.yaml.

3. Renew the certificates for the API server and related components:

    kubeadm certs renew admin.conf
    kubeadm certs renew apiserver
    kubeadm certs renew apiserver-kubelet-client
    kubeadm certs renew controller-manager.conf
    kubeadm certs renew front-proxy-client
    kubeadm certs renew scheduler.conf
   

4. Finally, restart the Kubelet service once more:

    systemctl restart kubelet
   

Step 5: Check the Health of the ETCD Cluster

Once the certificates have been renewed, verify the health of your ETCD cluster using etcdctl:

# Check ETCD health
ETCDCTL_API=3 etcdctl --endpoints=https://IP:PORT \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  endpoint health --cluster -w table

If you're using containerized ETCD, here's the workaround to check health:

# For containerized ETCD, run this command to check health
ctr run --rm -t --net-host --mount type=bind,src=/etc/kubernetes/pki/etcd,dst=/etc/kubernetes/pki/etcd,options=rbind:rw registry.k8s.io/etcd:3.5.6-0 test1 etcdctl \
  --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key \
  --cacert /etc/kubernetes/pki/etcd/ca.crt --endpoints https://IP:Port \
  endpoint health --cluster -w table

And that's how you renew your ETCD and Kubernetes certificates! Hopefully, this guide saves your next chill Friday. Happy clustering!

See you in the next blog post :)