Cross Account Access with EKS Pod Identity

Before we dive into Cross Account Pod Identity support, let's review some history. In the early days of EKS, if a pod needed to access another AWS resource, it had to use the Node IAM role. For example, if an app running in a pod needed to access an S3 bucket, the Node IAM role had to have the right IAM policy to access that specific S3 bucket.

However, this raised security concerns. Many of us, including myself, used KIAM and kube2iam to solve this problem by allowing granular access for Kubernetes applications. But managing these proxies wasn't easy either.

Then AWS IRSA (IAM roles for service accounts) simplified this complexity. However, for larger organizations with multiple AWS accounts, this also became a challenge because managing OIDC providers isn't easy. Before we dive in to EKS Pod Identity here is a simple understanding of KIAM, kube2iam and AWS IRSA.

EKS Pod Identity Launch

In 2023, AWS announced the launch of EKS Pod Identity, which simplifies the configuration of Kubernetes applications to obtain AWS IAM permissions. Unlike AWS IRSA, with EKS Pod Identity, we don't need the OIDC Provider dependency. You just need to install the EKS Pod Identity add-on and create the Pod Identity association. The diagram below explains the simple setup of EKS Pod Identity.

However, there are still some limitations with cross-account setup. There wasn't a direct way to do it, so we had to use some workarounds. We appreciate that AWS is great at listening to their customers.

Finally, AWS announced improvements for EKS Pod Identity cross-account access.

Internals of Cross-Account EKS Pod Identity

Let's break down how this works. In this scenario, Pods running in the EKS cluster within the Developer account can easily access resources in the Infrastructure account. This is done by having the IAM role in the Developer account assume a role in the Infrastructure account. It's as simple as that!

Let's explore two scenarios in this demo:

• As a Platform Engineer, Set up the external-dns add-on to create DNS records in Route53 across accounts.

• As a developer, learn how to access parameter store values from a different account.

Prerequisites:

• EKS Auto Mode Cluster with built-in EKS Pod Identity add-on. If you want to learn how to set up a EKS Auto Mode cluster, check my previous blog here.

• Two AWS Accounts

• Domain name configured in Route53

• Optional: ACM Certificate

Scenario 1: External DNS Cross-Account Access

We have an EKS Cluster running in the Developer Account, and the External DNS pods use the EKS Pod Identity to create DNS records in the Infrastructure Account.We already have a domain set up in Route53 within the Infrastructure Account.

Now let's create the IAM role and policies.

Infrastructure Account IAM

Include the IAM role ARN that we will create later in the developer account. Also, include the sts:ExternalId for more granular access..

Format for the sts:ExternalId

{AWS_Region}/{AWS_AccountNumber}/{EKS_Cluster_Name}/{NameSpace}/{ServiceAccountName}

For demonstration purposes, we are granting AmazonRoute53FullAccess access.

Create the IAM Role

Developer Account IAM

Skip adding permissions and create the IAM Role.

Go to the created IAM role and create an inline policy. Include the IAM role ARN that we previously created in the Infrastructure Account.

Create the policy

Install and configure the External DNS Addon in the EKS Cluster within the Developer Account.

Select External DNS

Select the EKS Pod Identity and choose the IAM Role we created. The External DNS add-on will be ready in a few minutes.

We needed to make one final configuration: edit the EKS Pod Identity Association for the External DNS.

Add the IAM role from the Infrastructure Account in the Target IAM Role section.

Now it's time to deploy a test application to check the External DNS Cross Account setup.

Here is the sample Kubernetes manifest we used for the application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: doggo-app
  labels:
    app: doggo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: doggo-app
  template:
    metadata:
      labels:
        app: doggo-app
    spec:
      containers:
      - name: doggo-app
        image: awsfanboy/doggo-app:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: doggo-app-service
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: doggo-app
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: eks-auto-alb
spec:
  controller: eks.amazonaws.com/alb
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: doggo-app-ingress
  annotations:
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-2:xxxxxxxxxx:certificate/{ACM}
spec:
  ingressClassName: eks-auto-alb
  rules:
  - host: app.infra.awsfanboy.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: doggo-app-service
            port:
              number: 80

Deploy the application

List the ingress object

Logs of the external-dns pod show that the DNS records were successfully created.

Route53 records were created successfully.

App is running successfully

We was able to set it up in just a few minutes, and it was very simple.

Scenario 2: App deployed on EKS Cluster with SSM Parameter access from Infrastructure account

We already have a parameter in SSM within the Infrastructure Account.

Now, let's create the IAM role and policies.

Infrastructure Account IAM

We are following the same steps we used for the External DNS add-on. Below are the screenshots for the Trust Relationship policy, IAM Policy of the the IAM Role to access the SSM Parameter store.

Developer Account IAM

We are following the same steps we used for the External DNS add-on. Below are the screenshots for the Trust Relationship policy, IAM Policy of the the IAM Role to assume the IAM role in the Infrastructure Account.

Create the EKS Pod Identity Association. Later, we are deploying a sample app into the default namespace with a Service Account named awsfanboy-sa.

Sample Kubernetes Manifest we used for deploying the sample application

apiVersion: v1
kind: ServiceAccount
metadata:
  name: awsfanboy-sa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: awsfanboy
  labels:
    app: awsfanboy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: awsfanboy
  template:
    metadata:
      labels:
        app: awsfanboy
    spec:
      serviceAccountName: awsfanboy-sa
      containers:
      - name: awscli
        image: amazon/aws-cli:latest
        command: ["sleep"]
        args: ["infinity"]

Let's deploy the sample app.

Now let's access the pod and run some commands to check the cross-account access.

• aws sts get-caller-identity: This shows that the identity is using the IAM role created in the Infrastructure Account.

• • 

• aws ssm get-parameter --name <parameter_name> The pod can access the parameter from the SSM Parameter Store in the Infrastructure Account.

  • 

Conclusion

We can clearly see that this new enhanced and streamlined way of EKS Pod Identity for cross-account access can be configured within a few seconds, reducing the workload for the EKS platform team and developers.

Reference

• https://aws.amazon.com/blogs/aws/amazon-eks-pod-identity-simplifies-iam-permissions-for-applications-on-amazon-eks-clusters/

• https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-eks-pod-identity/

• https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

• https://aws.amazon.com/blogs/containers/amazon-eks-pod-identity-streamlines-cross-account-access/