In Kubernetes version 1.26 or later, you might have encountered this frustrating error no matches for kind 'HorizontalPodAutoscaler' in version 'autoscaling/v2beta2'. This article will help you automatically fix the API version in your Helm release secrets.

TL;DR

Suddenly seeing "no matches for kind 'HorizontalPodAutoscaler' in version 'autoscaling/v2beta2'" or "no matches for kind 'HorizontalPodAutoscaler' in version 'autoscaling/v2beta1'" when running Helm upgrades? This happens because these API versions for HPA are deprecated in Kubernetes 1.26+. Use the command below to automatically update the API version in your Helm release secrets:

curl -sL https://gist.githubusercontent.com/nh4ttruong/172936eed756ca11e60efacf42117d2c/raw/31637ec5fed7a41ae202711304671df2de0bc1cd/fix-api-version.sh | bash -s

Always review scripts before piping them directly to bash. The script above updates Helm release secrets to use `autoscaling/v2` instead of deprecated API versions.

Problem

If you've recently upgraded your Kubernetes cluster to version 1.26 or later, you might have encountered this frustrating error "no matches for kind 'HorizontalPodAutoscaler' in version 'autoscaling/v2beta2'"

This error occurs because Kubernetes 1.26 has officially removed support for the autoscaling/v2beta2 API version, which was previously marked as deprecated. The current recommended version is `autoscaling/v2`

This issue is particularly problematic when dealing with Helm charts for several reasons:

• Embedded API Versions: Helm charts often have the API version hard-coded in their templates. When these charts were created with older API versions, they won't automatically adapt to newer Kubernetes versions.

• Release History Storage: Helm stores the deployed resources' state (including API versions) in secrets within your Kubernetes cluster. Even if you update your chart's templates, the previously deployed release information still contains references to the deprecated API.

• Update Mechanism Issues: When you attempt to upgrade a release with helm upgrade, Helm compares the new manifest against the stored release data. If it encounters incompatible API versions, the upgrade fails.

• No Automatic Migration: Helm doesn't automatically migrate resources from deprecated APIs to their replacements, requiring manual intervention.

The result is that even after updating your Helm chart's templates to use autoscaling/v2, you might still face errors because the stored release data references the now-removed autoscaling/v2beta2 API.

How to fix?

Here's a comprehensive guide to fixing this issue within your Helm releases:

1. Identify the Affected Helm Release

First, locate the secret that stores your Helm release configuration:

kubectl get secret -l owner=helm,status=deployed --all-namespaces

From the output, identify the secret related to your specific Helm release. It will typically follow a naming pattern that includes your release name.

2. Extract and Decode the Release Configuration

Export the secret to a file:

kubectl get secret <helm_release_secret> -n <namespace> -o yaml > release.yaml

Then decode the stored Helm release data:

cat release.yaml | grep -oP '(?<=release: ).*' | base64 -d | base64 -d | gzip -d > release.data.decoded

This command extracts the release data, decodes it from base64 (twice, as Helm doubly encodes it), and decompresses it.

3. Modify the HorizontalPodAutoscaler API Version

Open the decoded file in your preferred text editor:

nano release.data.decoded  # or vim, or any other editor

Search for all occurrences of autoscaling/v2beta2 and replace them with autoscaling/v2.

Be sure to verify that the structure of any HorizontalPodAutoscaler resources is compatible with the v2 API. In particular, note that the metrics specification might need adjustments.

4. Re-encode and Update the Helm Secret

After making your changes, you need to re-encode the file:

cat release.data.decoded | gzip | base64 | base64 | tr -d '\n' > encoded_release.txt

Create a patch file named patch.json:

{
  "data": {
    "release": "<NEW_ENCODED_RELEASE_DATA>"
  }
}

Replace <NEW_ENCODED_RELEASE_DATA> with the content of your encoded_release.txt file.

Apply the patch to update the Helm release secret:

kubectl patch secret <helm_release_secret> -n <namespace> --patch-file=patch.json

5. Verify and Redeploy

Finally, run a Helm upgrade to ensure the fix is applied:

helm upgrade <release_name> <chart_name> -n <namespace>

To verify that the HorizontalPodAutoscaler is now using the correct API version:

kubectl get hpa -n <namespace> -o yaml | grep apiVersion

This should show apiVersion: autoscaling/v2 for your HorizontalPodAutoscaler resources.

Prevention Measures

To avoid similar issues in the future:

1. Stay Informed About API Deprecations: Before upgrading Kubernetes, always check the release notes for any APIs that are deprecated and will be removed.

2. Update Your Helm Charts: Make sure your charts are regularly updated to support the latest Kubernetes API versions.

3. Use Automated Tools: Consider using tools like pluto to find deprecated API usage in your cluster before upgrading.

4. Test in Non-Production: Always test Kubernetes upgrades in a staging environment first to identify these kinds of issues.

5. Helm Chart Maintenance: For teams maintaining their own charts, set up a process to regularly review and update API versions.

Conclusion

API deprecations are a common challenge when upgrading Kubernetes, and Helm's release storage can make these issues especially difficult to solve. By learning how to diagnose and fix these issues within Helm's stored release data, you can ensure smoother upgrades and reduce disruptions to your services.

This specific fix for the HorizontalPodAutoscaler API version issue shows how important it is to stay updated with Kubernetes changes while keeping your deployment tools like Helm compatible.

Have you faced other interesting challenges when upgrading Kubernetes or working with Helm charts? Let me know.