Exploring Kubernetes Service Account Tokens and Secure Workload Identity Federation

Ever wonder how AWS IRSA or Azure AD workload identity works in Kubernetes?

How about GCP workload identity?

Well, imagine this… You go to an amusement park, get a ticket from the ticket booth, then you go to the front gate and you present your ticket to security folks at the gate. They scan your ticket, validate it and give you a wristband if your ticket is good, you might even get multiple wristbands if you paid for the extras. When you get into the park you can use your wristbands to access certain rides or VIP experiences, get something to eat if you have a meal plan, jump the queues etc...

Through the magic of service account token volume projection and OIDC authentication, Kubernetes workloads can authenticate in the cloud with an OIDC-compatible JSON Web Token (JWT) issued by the Kubernetes API server.

OpenID Connect (OIDC) authentication is a secure approach for verifying user identities in applications, leveraging trusted identity providers. OIDC builds upon the OAuth 2.0 protocol, providing a straightforward identity layer. It allows applications to establish a connection with various cloud providers OIDC identity providers like AWS IAM OIDC Identity Provider, Azure Managed Identity, and Google Cloud workload identity pool provider. In this setup, the cluster acts as the token issuer, generating tokens for authentication, while the cloud provider OIDC identity providers serve as the relying party, validating and authorizing access to resources.

Hi, I'd like to acquire a JWTicket to <insert-fav-cloud-provider> amusement park

Kubernetes offers a mechanism to specify desired properties of the token such as audience and expirationSeconds through ServiceAccount token volume projection.

$ kubectl create serviceaccount test
$ kubectl create token test #decode it @ jwt.io
$ kubectl create token test --audience "sts.amazon.com" #decode it @ jwt.io
$ kubectl create token test --duration 10m #decode it @ jwt.io
$ kubectl create token test --bound-object-kind Pod --bound-object-name testpod #decode it @ jwt.io

Before a relying party can validate the service account token, it needs to fetch the necessary information from the cluster's discovery document and JWKS (JSON Web Key Set). Here's a breakdown of the process:

1. Sending the Service Account Token: The relying party receives the service account token from the Kubernetes workload.

2. Retrieving the Discovery Document: The relying party extracts the "iss" (issuer) claim from the token, which represents the URL endpoint where the discovery document can be obtained. It fetches the discovery document from that URL.

3. Obtaining the JWKS: The discovery document contains information about the cluster's OIDC configuration, including the "jwks_uri" (JSON Web Key Set URI). The relying party retrieves the JWKS from this URI. The JWKS contains the public keys necessary to verify the token's signature.

4. Validating the Token: With the discovery document and JWKS in hand, the relying party can now validate the service account token. It verifies the token's signature using the public keys from the JWKS, checks the token's expiration and other claims, and ensures that the token is issued by a trusted issuer.

The Kubernetes API server publishes the discovery document at the /.well-known/openid-configuration endpoint and the JWKS at the /openid/v1/jwks endpoint. You can check the availability of the discovery document and JWKS in your Kubernetes cluster by accessing the respective URLs.

$ aws eks describe-cluster --name  --query "cluster.identity.oidc.issuer" --output text
$ curl ${ISSUER_URL}/.well-known/openid-configuration
$ curl ${ISSUER_URL}/openid/v1/jwks

$ az aks show --resource-group  --name  --query "oidcIssuerProfile.issuerUrl" -o tsv
$ curl {ISSUER_URL}/.well-known/openid-configuration
$ curl {ISSUER_URL}/openid/v1/jwks

# view the oidc documents
$ kubectl proxy -p 8999
# get the discovery document
$ curl http://127.0.0.1:8999/.well-known/openid-configuration
# get the JWKS document
$ curl http://127.0.0.1:8999/openid/v1/jwks

If you have an on-prem cluster with an issuer URL that is not accessible over the internet, you'll need to cache and serve these documents from an internet-accessible URL; you can use any cloud provider's object storage. Update the API server with the new issuer URL by setting the --service-account-issuer flag.

• --service-account-issuer - a valid url capable of serving OpenID discovery documents at <issuer-url>/.well-known/openid-configuration.

• --service-account-jwks-uri - value should be <issuer-url>/openid/v1/jwks.

• I'll be using use a KIND cluster for this demo

# I'm using AWS S3 to cache and serve these documents
# you can use cloud object storage
# be sure to CHANGE the bucket name
$ aws s3api create-bucket --bucket satp-demo-xyz \
  --object-ownership BucketOwnerPreferred \
  --create-bucket-configuration LocationConstraint=ca-central-1 \
  --region ca-central-1

$ aws s3api put-public-access-block \
  --bucket satp-demo-xyz \
  --public-access-block-configuration '{
      "BlockPublicAcls": false,
      "IgnorePublicAcls": false,
      "BlockPublicPolicy": false,
      "RestrictPublicBuckets": false
    }'

$ ISSUER_URL="satp-demo-xyz.s3.ca-central-1.amazonaws.com"

# create kind cluster yaml, set the 
$ tee kind-conf.yaml <<EOF
  kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  nodes:
  - kubeadmConfigPatches:
    - |
       kind: ClusterConfiguration
       apiServer:
          extraArgs:
            service-account-issuer: https://${ISSUER_URL}
            service-account-jwks-uri: https://${ISSUER_URL}/openid/v1/jwks
EOF

$ kind create cluster --config kind-conf.yaml

# retrieve the discovery and jwks document
$ kubectl proxy -p 8999

$ curl http://127.0.0.1:8999/.well-known/openid-configuration | jq > discovery.json

$ curl http://127.0.0.1:8999/openid/v1/jwks | jq > keys.json

# upload the documents to S3
$ aws s3 cp discovery.json \
  s3://satp-demo-xyz/.well-known/openid-configuration \
  --acl public-read

$ aws s3 cp keys.json \
  s3://satp-demo-xyz/openid/v1/jwks \
  --acl public-read

# verify that the documents are accessible
$ curl https://${ISSUER_URL}/.well-known/openid-configuration

$ curl https://${ISSUER_URL}/openid/v1/jwks

Here's my ticket, can I have a wristband?

We have a ticket now, security will need to validate our ticket to determine if we get wristbands, but how does security validate the ticket?

Kubernetes workloads can securely access cloud resources by leveraging the Security Token Service (STS) provided by the cloud provider. STS supports identity federation, allowing Kubernetes workloads to exchange their existing authentication tokens for temporary security credentials. When a request is made to STS, the token is validated to ensure its authenticity and integrity. If the token is valid, STS generates temporary security credentials, which are then returned to the Kubernetes workload. These temporary credentials provide authorized access to cloud resources, enabling Kubernetes workloads to interact with various services and APIs within the cloud environment.

Let's configure our relying party to validate our token.

AWS specific implementation

• create an IAM OIDC IdP and configure it to trust our Kubernetes cluster (acting as an OIDC-compatible IdP) and your AWS account.

• create an IAM role for the IdP with a trust policy that allows identities authenticated by the OIDC provider to assume the role. In the trust policy conditions, the audience is sts.amazonaws.com, the subject will be our Kubernetes service account URN system:serviceaccount:<namespace>:<name>.

• attach policies to the IAM role for access.

# follow the steps here to get the SHA1 thumbprint: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
# create IAM OIDC IdP
$ aws iam create-open-id-connect-provider \
  --url https://$ISSUER_URL \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list ABCDEFGHIJKLMOPQRSTUVWXYZ123456789QWERTY

# GET the OIDC IdP arn
$ OIDC_ARN=$(aws iam list-open-id-connect-providers \
  --query "OpenIDConnectProviderList[? contains(Arn,'${ISSUER_URL}')].Arn" --output text)

$ echo $OIDC_ARN

# create AWS trust policy document
$ tee trust-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::146335578023:oidc-provider/satp-demo-xyz.s3.ca-central-1.amazonaws.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "satp-demo-xyz.s3.ca-central-1.amazonaws.com:aud": "sts.amazonaws.com",
                    "satp-demo-xyz.s3.ca-central-1.amazonaws.com:sub": "system:serviceaccount:aws:default"
                }
            }
        }
    ]
}
EOF

$ aws iam create-role \
  --role-name satp-demo-role \
  --assume-role-policy-document file://trust-policy.json

$ aws iam attach-role-policy \
  --role-name satp-demo-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

$ AWS_IAM_ROLE_ARN=$(aws iam get-role --role-name satp-demo-role \
  --query "Role.Arn")

$ echo $AWS_IAM_ROLE_ARN

Azure implementation

• create a user-assigned managed identity and configure a federated identity credential. The audience is api://AzureADTokenExchange and the subject will be our Kubernetes service account URN system:serviceaccount:<namespace>:<name>.

• grant the MI access to Azure resources.

$ RESOURCE_GROUP=satp-demo LOCATION=canadacentral

$ AZURE_TENANT_ID="$(az account show --query tenantId -otsv)"

$ SUBSCRIPTION_ID="$(az account show --query id -otsv)"

$ az group create --name "${RESOURCE_GROUP}" --location "${LOCATION}"

$ az identity create --name "${RESOURCE_GROUP}-uami" \
  --resource-group "${RESOURCE_GROUP}"

$ UAMI_OBJECT_ID=$(az identity show --name "${RESOURCE_GROUP}-uami" \
  --resource-group "${RESOURCE_GROUP}" --query 'principalId' -otsv)

$ UAMI_CLIENT_ID=$(az identity show --name "${RESOURCE_GROUP}-uami" \
  --resource-group "${RESOURCE_GROUP}" --query 'clientId' -otsv)

$ az role assignment create --role "Contributor" \
  --scope /subscriptions/$SUBSCRIPTION_ID \
  --assignee $UAMI_OBJECT_ID 

$ az identity federated-credential create \
  --name "kubernetes-federated-credential" \
  --identity-name "${RESOURCE_GROUP}-uami" \
  --resource-group "${RESOURCE_GROUP}" \
  --issuer "https://${ISSUER_URL}" \
  --subject "system:serviceaccount:azure:default"

# create a keyvault and grant the MI access
$ KV_NAME=${RESOURCE_GROUP}-test-kv
$ az keyvault create --resource-group "${RESOURCE_GROUP}" \
  --location "${LOCATION}" \
  --name "${KV_NAME}"

$ az keyvault secret set --vault-name "${KV_NAME}" \
  --name "test" \
  --value "Hello\!"

$ az keyvault set-policy --name "${KV_NAME}" \
  --secret-permissions get \
  --object-id "${UAMI_OBJECT_ID}"

GCP Implementation

• create a Google service account, a workload identity pool and a provider.

• configure the workload identity pool provider to trust our cluster and validate the tokens

• add iam policy bindings to the service account resource to allow the members of our workload identity pool to run operations as the service account.

• add iam policy bindings to your project to grant the service account access to resources within the project.

• grant the service account additional roles for access

$ PROJECT_NAME=satp-demo

$ gcloud projects create $PROJECT_NAME

$ gcloud config set project $PROJECT_NAME

$ PROJ_NUMBER=$(gcloud projects describe $PROJECT_NAME --format json | jq .projectNumber)

$ gcloud services enable iamcredentials.googleapis.com iam.googleapis.com cloudresourcemanager.googleapis.com

$ GSA_NAME=$PROJECT_NAME-gsa

$ gcloud iam service-accounts create $GSA_NAME \
  --display-name="satp demo gsa" \
  --description="google service account to be impersonated"

$ GSA_EMAIL="${GSA_NAME}@${PROJECT_NAME}.iam.gserviceaccount.com"

$ gcloud projects add-iam-policy-binding $PROJECT_NAME \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/iam.serviceAccountTokenCreator"

$ gcloud projects add-iam-policy-binding $PROJECT_NAME \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/serviceusage.serviceUsageViewer"

$ gcloud iam workload-identity-pools create satp-demo-pool \
  --location="global" \
  --description="k8s service account token projection demo pool" \
  --display-name="satp demo pool"

$ gcloud iam workload-identity-pools providers create-oidc satp-demo-pool-provider \
  --location="global" \
  --workload-identity-pool="satp-demo-pool" \
  --issuer-uri="https://${ISSUER_URL}" \
  --attribute-mapping="google.subject=assertion.sub"

$ WIP_NAME=$(gcloud iam workload-identity-pools list --location=global --format='value(name)')

$ SA_MEMBER="iam.googleapis.com/${WIP_NAME}/subject/system:serviceaccount:gcp:default"

$ gcloud iam service-accounts add-iam-policy-binding $GSA_EMAIL \
  --member="principal://${SA_MEMBER}" \
  --role=roles/iam.workloadIdentityUser

$ WIPP_NAME=$(gcloud iam workload-identity-pools providers list \
  --workload-identity-pool=satp-demo-pool \
  --location=global --format='value(name)')

$ gcloud iam workload-identity-pools create-cred-config $WIPP_NAME \
  --service-account=$GSA_EMAIL \
  --credential-source-file="/var/run/secrets/gcp/token" \
  --output-file="gcp-cred-config.json"

Let's check out some of the cool rides in the park

Let's configure some pods and try to access some resources in AWS, Azure, and GCP.

AWS

$ kubectl create namespace aws

$ tee aws.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: satp-demo
  namespace: aws
spec:
  containers:
  - image: amazon/aws-cli:latest
    name: test
    command: ['sh', '-c', 'sleep 10000']
    env:
    - name: AWS_DEFAULT_REGION
      value: "ca-central-1"
    - name: AWS_ROLE_ARN
      value: ${AWS_IAM_ROLE_ARN}
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/aws/token
    - name: AWS_ROLE_SESSION_NAME
      value: default
    volumeMounts:
    - mountPath: /var/run/secrets/aws
      name: aws-identity-token
      readOnly: true
  volumes:
  - name: aws-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 360
          path: token
EOF

$ kubectl apply -f aws.yaml
# exec into the pod
$ kubectl exec -it pod/satp-demo -n aws -- /bin/sh

$ aws sts get-caller-identity

$ aws s3 ls

Azure

$ kubectl create namespace azure

$ tee azure.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: satp-demo
  namespace: azure
spec:
  containers:
  - image: mcr.microsoft.com/azure-cli:latest
    name: test
    command: ['sh', '-c', 'sleep 10000']
    env:
    - name: AZURE_AUTHORITY_HOST
      value: https://login.microsoftonline.com/
    - name: AZURE_CLIENT_ID
      value: ${UAMI_CLIENT_ID}
    - name: AZURE_TENANT_ID
      value: ${AZURE_TENANT_ID}
    - name: AZURE_FEDERATED_TOKEN_FILE
      value: /var/run/secrets/azure/token
    - name: VAULT_NAME
      value: ${KV_NAME}
    volumeMounts:
    - mountPath: /var/run/secrets/azure
      name: azure-identity-token
      readOnly: true
  volumes:
  - name: azure-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: api://AzureADTokenExchange
          expirationSeconds: 3600
          path: token
EOF

$ kubectl apply -f azure.yaml

$ kubectl exec -it pod/satp-demo -n azure -- /bin/sh

$ az login --service-principal -u ${AZURE_CLIENT_ID} \
  -t {$AZURE_TENANT_ID} \
  --federated-token $(cat $AZURE_FEDERATED_TOKEN_FILE)

$ az identity list

$ az keyvault secret show --name test --vault-name $VAULT_NAME

GCP

$ kubectl create namespace gcp

$ tee gcp.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: satp-demo
  namespace: gcp
spec:
  containers:
  - image: gcr.io/google.com/cloudsdktool/google-cloud-cli:alpine
    name: test
    command:
      - sh
      - -c
      - |
        gcloud iam workload-identity-pools \\
        create-cred-config $WIPP_NAME \\
        --service-account=$GSA_EMAIL \\
        --credential-source-file='/var/run/secrets/gcp/token' \\
        --output-file=\$GOOGLE_APPLICATION_CREDENTIALS 
        sleep 100000
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: gcp-cred-config.json
    - name: CLOUDSDK_COMPUTE_REGION
      value: northamerica-northeast2
    - name: WIPP_NAME
      value: ${WIPP_NAME}
    - name: GSA_EMAIL
      value: ${GSA_EMAIL}
    volumeMounts:
    - mountPath: /var/run/secrets/gcp
      name: gcp-identity-token
      readOnly: true
  volumes:
  - name: gcp-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: https://iam.googleapis.com/${WIPP_NAME}
          path: token
EOF
$ kubectl apply -f gcp.yaml

$ kubectl exec -it pod/satp-demo -n gcp -- /bin/sh

$ gcloud auth login --cred-file=$GOOGLE_APPLICATION_CREDENTIALS

$ gcloud config set project satp-demo

$ gcloud services list

$ gcloud projects list

We've just barely scratched the surface but should have a better understanding of how to securely streamline authentication and authorization of Kubernetes workloads in the cloud. With open-source projects like AWS amazon-eks-pod-identity-webhook, Azure's azure-workload-identity, or gcp-workload-identity-federation-webhook by pfnet-research you can implement this at scale.

By combining OAuth's authorization capabilities with OIDC's authentication as we've done with our implementation of workload Identity federation, we can secure and streamline authentication and authorization workflows, making it easier to access the cloud from your Kubernetes workload. This streamlined process eliminates secret management burdens and improves security, enabling scalable applications in multi-cloud and hybrid environments.

Enjoy the amusement park of cloud computing!

Reference materials:

• https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection

• https://darutk.medium.com/diagrams-of-all-the-openid-connect-flows-6968e3990660

• https://azure.github.io/azure-workload-identity/docs/

• https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

• https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers

• https://cloud.google.com/iam/docs/reference/sts/rest/v1/TopLevel/token

• https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html

• https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html

• https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow#third-case-access-token-request-with-a-federated-credential

• https://web-identity-federation-playground.s3.amazonaws.com/index.html