Mastering AWS EKS Cluster Upgrades: Best Practices, Prerequisites, and Key Considerations

Upgrading an Amazon Elastic Kubernetes Service (EKS) cluster ensures you stay current with Kubernetes features, security patches, and API deprecations. However, the process requires careful planning to prevent service disruptions. This detailed blog will guide you through the prerequisites, key considerations, and best practices to ensure a smooth upgrade.

---

Key Points to Remember Before Starting

1. Maximum of 5 Available IPs per Node:
   Ensure each node in the cluster has at least 5 available IP addresses in its subnet. A shortage of IPs during the upgrade can cause nodes to fail in joining the cluster.

2. Downgrade Is Not Supported:
   Kubernetes upgrades on AWS EKS are irreversible. Once a cluster is upgraded to a new version, you cannot roll back to a previous version.

3. Check Kubernetes Release Notes:
   Always review the Kubernetes release notes for deprecations, feature changes, and breaking updates in the target version.

4. Lower Version Upgrades Are Not Possible:
   AWS EKS does not allow upgrading to a lower Kubernetes version. If you need a lower version, you must deploy a new cluster and migrate workloads manually.

5. Sequential Upgrades Only:
   EKS only supports upgrading one minor version at a time. For example, if your cluster is on version 1.25, you must upgrade to 1.26 before moving to 1.27.

---

Prerequisites for AWS EKS Cluster Upgrades

1. Check Cluster Version Compatibility

• Use the AWS CLI to determine your current Kubernetes version:

    aws eks describe-cluster --name <cluster-name> --query cluster.version
  

• Confirm the target version is supported by AWS EKS. Sequential upgrades are mandatory; skipping versions is not allowed.

2. Backup the Cluster

• Export all resources and configurations to YAML:

    kubectl get all --all-namespaces -o yaml > cluster-backup.yaml
  

• Use tools like Velero for automated snapshots, including PVCs and critical data.

3. Validate Subnet IP Availability

• Ensure each node’s subnet has at least 5 available IP addresses:

    aws ec2 describe-subnets --subnet-ids <subnet-id> --query "Subnets[].AvailableIpAddressCount"
  

4. Review Application Compatibility

• Identify and update deprecated APIs:

    kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found
  

• Test workloads on a staging cluster with the target Kubernetes version.

5. Update Kubernetes Tools

• Ensure kubectl, Helm, and other tools match the target version:

    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
    chmod +x kubectl
    sudo mv kubectl /usr/local/bin/
    kubectl version --client
  

6. Upgrade Node Groups

• Managed Node Groups:

    aws eks update-nodegroup-version --cluster-name <cluster-name> --nodegroup-name <nodegroup-name>
  

• Self-Managed Nodes:

  • Update to the latest AMI for your region from EKS Optimized AMIs.

  • Replace older nodes with updated ones using a rolling update strategy.

7. Check IAM Permissions

• Verify that your IAM role includes permissions like:

  • eks:UpdateClusterVersion

  • eks:UpdateNodegroupVersion

8. Update Critical Add-ons

• CoreDNS:

    kubectl apply -f https://github.com/coredns/deployment/kubernetes/coredns.yaml
  

• kube-proxy:

    aws eks update-addon --cluster-name <cluster-name> --addon-name kube-proxy
  

• VPC CNI Plugin:

    aws eks update-addon --cluster-name <cluster-name> --addon-name vpc-cni
  

9. Plan for Downtime

• Inform stakeholders and schedule upgrades during off-peak hours. Some workloads may experience brief disruptions.

---

Steps to Perform the Upgrade

1. Upgrade the Control Plane

   • Use the AWS CLI to upgrade the EKS control plane:

       aws eks update-cluster-version --name <cluster-name> --kubernetes-version <target-version>
     

2. Upgrade Node Groups

   • Managed Node Groups:

       aws eks update-nodegroup-version --cluster-name <cluster-name> --nodegroup-name <nodegroup-name>
     

   • Self-Managed Nodes: Replace with nodes running the updated AMI.

3. Validate Post-Upgrade

   • Confirm the cluster’s health and ensure workloads are functioning as expected:

       kubectl get nodes
       kubectl get pods --all-namespaces
     

---

Special Note on Downgrades and Lower Version Upgrades

• Downgrades: Kubernetes upgrades on AWS EKS are irreversible. Always test upgrades in a staging environment to avoid issues.

• Lower Version Upgrades: If you need to migrate to a lower Kubernetes version, the only option is to create a new EKS cluster with the desired version and migrate workloads manually.

---

Conclusion

Upgrading an AWS EKS cluster requires meticulous preparation. Ensuring application compatibility, sufficient IP availability, and thorough backup practices are critical to a successful upgrade. Always stay informed about release notes and plan for contingencies to minimize downtime.

Would you like to automate any of these processes? Let me know in the comments!