From Base64 to Fort Knox: Fortifying Kubernetes Secrets

Today, let's delve into the internal workings of Kubernetes secrets. When you create a secret in a Kubernetes environment, do you believe it's entirely safe and secure?

The answer is no. While it's an improvement over storing secrets in plain text files, it's important to note that anyone with access to these secrets can view and decode them. Essentially, secrets utilize Base64 encoding to protect sensitive information.

On the Linux platform, you can use the following command to encode any string to Base64:

echo -n "YourTextHere" | base64

and Here's the decode command.

echo <Your-encoded-string> | base64 --decode

Encoded secrets are stored in your etcd database, which Kubernetes utilizes. It's critical to refrain from pushing secret files to any source code management systems. Now, let's explore how we can access data within the etcd database.

Ensure that etcdctl is installed on your master node. For installation instructions, refer to the etcd-install-link for the latest or a specific version of your choice.

Firstly, set the etcd API version to 3:

ETCDCTL_API=3

Verify the presence of ca.crt, server.crt, and server.key certificates and keys for authentication at this location:

ls /etc/kubernetes/pki/etcd/

Now, execute the following command, ensuring to replace <Your-secret-name> with your secret's name:

etcdctl \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ get /registry/secrets/default/<Your-secret-name> | hexdump -C

You may be able to view your secret here. It might be challenging to locate, but it will be visible. However, this visibility is not secure. No one wants their secrets exposed like this. So, how can we secure these secrets? This is where encryption at rest becomes crucial.

To determine if secret encryption is configured, check /etc/kubernetes/manifests/kube-apiserver.yaml for the presence of --encryption-provider-config. If it's not configured, let's proceed to set it up.

As depicted in the image above, you have the option to select which resources you wish to secure. However, our specific requirement is to safeguard only secrets. Consequently, we will exclusively opt for securing secrets. Additionally, in the configuration displayed, the 'providers' list indicates that 'identity' occupies the top position. This suggests that the resources listed are not encrypted at all. The remaining encryption details are provided in the image below.

Here, you can find further details about this topic: Kubernetes - Encrypt Data

To encrypt a secret, it's necessary to remove the 'identity' encryption first, allowing encryption of secrets. Next, select your preferred encryption method. In my case, I'll use 'aescbc.' This encryption method requires a random key, which we can generate using the command:

head -c 32 /dev/urandom | base64

Place this randomly generated value in a file named 'enc.yaml' and relocate it to '/etc/kubernetes/enc/' for better file management. Now, let's modify the 'kube-apiserver.yaml' file to implement this encryption configuration. Add the following line, as depicted in the image, ensuring to specify the volume and volume mounts.

After applying these modifications, the kube-apiserver might undergo a restart, which could take some time. If you're wondering why we're dealing with a YAML file in this location without using the 'kubectl apply' command, I recommend reading this concise blog post about static pods: What Are the Static Pods on Kubernetes.

Once the kube-apiserver restarts, create a new secret containing a specific value. Then, utilize the previously mentioned 'etcdctl' command to verify the existence of the new secret in etcd. You'll notice a surprising change – the secret value will no longer be accessible. But, what about previously created secrets? These were not encrypted earlier. To encrypt them, use the following command:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Voilà! The secrets are now much more secure than before, as they're no longer solely relying on base64 encoding.

For a more comprehensive understanding, refer to the detailed documentation: Kubernetes - Encrypt Data.

Thanks for reading,