Mounting AWS Secrets as Volumes in EKS

A coworker asked me what I meant in my last blog post when I said I was leveraging a service-role mapping to inject secrets from AWS Secrets Manager into my production pods. In this post, I'll explain what and how.

tl;dr: I have a GitHub repo that shows the OpenTofu bits of this blog post, as well as a sample Deployment yaml file for the Kubernetes bits. Just know that you still have to install the Secrets Store CSI Driver in your cluster (covered in the first section below).

The Secrets Store CSI Driver

What's a CSI? It's a Container Storage Interface, which means it acts as a mechanism to attach data resources as volumes. The Secrets Store CSI Driver was created by the Auth Special Interest Group in the Kubernetes Community to allow for the better handling of secrets as volumes.

The best part? It can automatically handle secrets rotation!!! However, this isn't enabled by default, and the feature is still in alpha, so there's a risk of that feature changing in future releases. The installation instructions below assume that you'll want that feature activated.

Most of what's written in the next section can be found by clicking through the various pages of their docs, so consider this just a consolidated view. If you want to get to the actual usage, skip to the following section.

Installation for AWS EKS

Helm is one of my favorite things, and I'm reminded of that fact any time a tool I want to use has a ready-to-go Helm chart. These steps assume that you have it installed.

And the Secrets Store CSI Driver does. With your kubectl context set to the Kubernetes cluster you'd like to target, run the following commands:

# add the SSCD chart repo to Helm's cached list of charts
$ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

# insall the chart into the kube-system namespace
$ helm install -n kube-system \
  csi-secrets-store \
  secrets-store-csi-driver/secrets-store-csi-driver \
  # this is where we set secrets rotation options
  --set enableSecretRotation=true \
  --set rotationPollInterval=3000s

AWS doesn't provide a Helm chart for their secrets store provider, but it is a one-line installation, so not a big deal:

# install the AWS provider for the secrets store CSI driver
$ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

That covers installation, so how do we use it?

Create Secrets and Policies with OpenTofu

For your Pod to access resources in AWS SecretsStore, its assigned ServiceAccount has to have access.

If we pick up where the last blog post left off, we have some HCL that looks like this condensed version:

# IAM Role
resource "aws_iam_role" "service_role" {
  name = var.service_account_name
  assume_role_policy = jsonencode({
    # ...
  })
}

# K8s ServiceAccount
resource "kubernetes_manifest" "service_account" {
  manifest = {
    # ...
  }
}

We're going to add five additional resources:

• a random string to use as a password

• an AWS SecretsManager secret

• a secret version that defines the values stored in that secret

• an IAM policy

• a policy-role binding

Let's start with the first three, using the built-in random_password resource:

# generating the password
resource "random_password" "db_password" {
  length           = 16 # feel free to change this or set a var
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?" # AWS-safe special chars
}

# creating the Secret were we'll store the password
resource "aws_secretsmanager_secret" "secret" {
  name  = "${var.service_account_name}_iam_secret"
}

# actually storing the password value in the Secret
resource "aws_secretsmanager_secret_version" "secret" {
  secret_id = aws_secretsmanager_secret.secret.id
  secret_string = jsonencode(
    merge(
      tomap({}),
      tomap({ "DB_PASS" = random_password.db_password.result })
    )
  )
}

The IAM policy will allow the basic read operations against the Secret ARN:

resource "aws_iam_policy" "secret_access_policy" {
  name        = "${var.service_account_name}-access-policy"
  description = "Access policy for the ${var.service_account_name} secret"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [{
      "Effect" : "Allow",
      "Action" : ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource" : [aws_secretsmanager_secret.secret.arn]
    }]
  })
}

Except, at this point, the policy isn't tied to any IAM identity yet. To apply this policy to our application, we'll need to bind it to the IAM role that the ServiceAccount is going to use:

resource "aws_iam_role_policy_attachment" "secret_policy_attachment" {
  role       = aws_iam_role.service_role.name
  policy_arn = aws_iam_policy.secret_access_policy.arn
}

Once you tofu apply (or terraform apply)these changes, all of the AWS bits will be taken care of. At this point, it's a matter of applying the right Kubernetes configurations.

Kubernetes!

Remember that Secrets Store CSI Driver we installed in the first section? It installed a CRD called SecretProviderClass in the cluster, which is what tells the cluster how to find the secrets that need to be attached as a volume.

Since all of the information I need already lives in my OpenTofu configs, I include this in there as well:

resource "kubernetes_manifest" "spc" {
  manifest = {
    "apiVersion" = "secrets-store.csi.x-k8s.io/v1alpha1" # the API the SSCD installed
    "kind"       = "SecretProviderClass" # our new CRD
    "metadata" = {
      "name"      = "${var.service_account_name}-aws-secret" # we'll need this in a minute
      "namespace" = var.target_namespace
    }
    "spec" = {
      "provider" = "aws" # the AWS provider we installed in the first section
      "parameters" = {
        # the secret we created in the second section
        "objects" = <<EOF
            - objectName: "${aws_secretsmanager_secret.secret.name}" 
              objectType: "secretsmanager"
        EOF
      }
    }
  }
}

After that, there's not much left. We just have to create a volume using the keys and values described in the Secrets Store CSI usage docs.

In your Deployment.spec.template.spec (or Job, Cronjob, whatever), add the following block of YAML to define your new volume:

volumes:
- name: <service-name>-secrets-store
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      # notice this matches the SPC we created above!
      secretProviderClass: "<service-account-name>-aws-secret"

From there, you just attach it to your main container the way you would any other volume:

volumeMounts:
- name: <service-name>-secrets-store
  mountPath: "/secrets/iam"
  readOnly: true

The one thing that I don't like is that it will use the name of the secret in AWS as the file name. So if you named your secret <service-account-name>_iam_secret like what's shown in the code block in the second section above, and you used the mount path /secrets/iam, then your secret data will be found at /secrets/iam/<service-account-name>_iam_secret. Despite there not being any extension on that file, its contents are JSON, so be prepared to parse whatever you read in:

{
  "DB_PASS": <the tofu-generated random string>
}

Sorry this one was so long again. I'll try to slim down the content in the future.