Using AWS Secrets Manager and Kubernetes Secrets Store CSI Driver for Secure Secret Management in EKS

AWS Secrets Manager is used to securely store and retrieve sensitive information for the application running on an EKS cluster. Here's how it works step-by-step:

1. AWS Secrets Manager Storage:

   • Secrets, such as database credentials, API keys, and environment-specific variables, are stored securely in AWS Secrets Manager.

   • In the SecretProviderClass section, the objects parameter specifies the objectName (secret in Secrets Manager) and how the secret's keys are mapped (jmesPath). For example, objectName: applicationname refers to a secret stored in Secrets Manager, and the nested jmesPath maps the paths within that secret to alias names used in the Kubernetes environment.

2. CSI Driver for Secrets Store:

   • The Kubernetes deployment uses the Secrets Store CSI Driver, which enables the Kubernetes pod to directly fetch secrets from AWS Secrets Manager.

   • The volumes section in the Deployment uses the CSI driver (secrets-store.csi.k8s.io) to mount the secrets as a volume. The SecretProviderClass specifies how the secrets should be fetched from AWS Secrets Manager.

3. Secrets Injection into the Pod:

   • When the pod is created, the Secrets Store CSI Driver connects to AWS Secrets Manager using the provided IAM role (eks.amazonaws.com/role-arn in the ServiceAccount).

   • The secrets defined in the SecretProviderClass (applicationname-secrets) are fetched and made available in the pod. They are either mounted as files at the specified mountPath (/mnt/secrets) or directly injected into environment variables.

4. Environment Variables from Secrets:

   • In the Deployment YAML, the environment variables for the applicationname container are defined using the valueFrom attribute. This references the secrets provided by the CSI driver.

   • Each environment variable references a key from the applicationname-secrets that the CSI driver fetched from AWS Secrets Manager.

   • For example:

       - name: DATABASE_HOST
         valueFrom:
           secretKeyRef:
             name: applicationname-secrets
             key: DATABASE_HOST
     

   • This means that the environment variable DATABASE_HOST is populated with the value of DATABASE_HOST key stored in the applicationname-secrets.

5. IAM Role for Secret Access:

   • The ServiceAccount named applicationname has an annotation (eks.amazonaws.com/role-arn) pointing to an IAM role (arn:aws:iam::xxxxxxxxxxx:role/applicationname-secrets).

   • This IAM role must have permissions to access the specified secrets in AWS Secrets Manager. When the pod is started, it assumes this IAM role to access the secrets.

     ---

Auto rotation of mounted contents and synced Kubernetes Secrets

When the secret/key is updated in external secrets store after the initial pod deployment, the updated secret will be periodically updated in the pod mount and the Kubernetes Secret.

Depending on how the application consumes the secret data:

1. Mount Kubernetes secret as a volume: Use auto rotation feature + Sync K8s secrets feature in Secrets Store CSI Driver, application will need to watch for changes from the mounted Kubernetes Secret volume. When the Kubernetes Secret is updated by the CSI Driver, the corresponding volume contents are automatically updated.

2. Application reads the data from container’s filesystem: Use rotation feature in Secrets Store CSI Driver, application will need to watch for the file change from the volume mounted by the CSI driver.

3. Using Kubernetes secret for environment variable: The pod needs to be restarted to get the latest secret as environment variable.

4. Use something like Reloader to watch for changes on the synced Kubernetes secret and do rolling upgrades on pods

Enable auto rotation

NOTE: This alpha feature is not enabled by default.

To enable auto rotation, enable the --enable-secret-rotation feature gate for the secrets-store container in the Secrets Store CSI Driver pods. The rotation poll interval can be configured using --rotation-poll-interval. The default rotation poll interval is 2m. If using helm to install the driver, set enableSecretRotation: true and configure the rotation poll interval by setting rotationPollInterval. The rotation poll interval can be tuned based on how frequently the mounted contents for all pods and Kubernetes secrets need to be resynced to the latest.

• The Secrets Store CSI Driver will update the pod mount and the Kubernetes Secret defined in secretObjects of SecretProviderClass periodically based on the rotation poll interval to the latest value.

• If the SecretProviderClass is updated after the pod was initially created

  • Adding/deleting objects and updating keys in existing secretObjects - the pod mount and Kubernetes secret will be updated with the new objects added to the SecretProviderClass.

  • Adding new secretObject to the existing secretObjects - the Kubernetes secret will be created by the controller.

        resource "helm_release" "secrets-csi-driver" {
          name = "secrets-store-csi-driver"

          repository = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"
          chart = "secrets-store-csi-driver"
          namespace = "kube-system"
          version = "1.4.3"

          set {
            name  = "syncSecret.enabled"
            value = true
          }
          set {
            name  = "enableSecretRotation"
            value = true
          }
          set {
            name  = "rotationPollInterval"
            value = "60s"
          }
        }

---

Reloader

Problem

We would like to watch if some change happens in ConfigMap and/or Secret; then perform a rolling upgrade on relevant DeploymentConfig, Deployment, Daemonset, Statefulset and Rollout

Solution

Reloader can watch changes in ConfigMap and Secret and do rolling upgrades on Pods with their associated DeploymentConfigs, Deployments, DaemonsetsStatefulsets and Rollouts.

secret

To perform rolling upgrade when change happens only on specific secrets use below annotation.

For a Deployment called foo have a Secret called foo-secret. Then add this annotation to main metadata of your Deployment

    kind: Deployment
    metadata:
      annotations:
        secret.reloader.stakater.com/reload: "foo-secret"
    spec:
      template: 
        metadata:

Use comma separated list to define multiple secrets.

    kind: Deployment
    metadata:
      annotations:
        secret.reloader.stakater.com/reload: "foo-secret,bar-secret,baz-secret"
    spec:
      template: 
        metadata:

Deploying Reloader to Kubernetes

    resource "helm_release" "reloader" {
      name       = "reloader"
      repository = "https://stakater.github.io/stakater-charts"  
      chart      = "reloader"
      namespace  = "kube-system"
      version    = "1.1.0"

        set {
        name  = "reloader.watchGlobally"
        value = "true"
      }
    }

Refrenece :
https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver.html
https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html
https://github.com/stakater/Reloader?tab=readme-ov-file

---

Stay Tuned!

Be sure to follow and subscribe for more updates and upcoming blogs.