While working on my pet project Shortlist, I needed to send emails. While I could have used a transactional email service like Mandrill, I wanted something cloud-based that could be configured with infrastructure-as-code. I couldn’t find any email service on Google Cloud, but AWS Simple Email Service (SES) struck me as a solid option. The main obstacle was authorising the API calls from workloads running in Google Kubernetes Engine (GKE) with the SES service. Yes, I could have created some long-lived credentials and made them available in Kubernetes via a secrets vault, but I wanted to follow best practice. To my knowledge, the best practice is for cloud workloads to assume IAM roles and authorise API calls using short-lived credentials.

Solution Outline

If I were working only within AWS, I might have chosen Pod Identity to allow my workload to assume a role that permits it to send emails using SES. Since this particular case spans two cloud providers, I went with a common standard implemented by both: OpenID Connect (OIDC). Both Google Cloud’s Workload Identity Federation and AWS’ IAM Roles for Service Accounts rely on the Kubernetes cluster implementing an OIDC provider that maps ServiceAccount resources to identities that can be used to federate. The OIDC identity tokens are mounted on containers using projected volumes, where they can be read by the AWS CLI or SDKs like boto3.

At a high level, I needed the following to get the end-to-end email flow up and running:

Code

Most of the relevant code here was written in HCL and run using OpenTofu. There’s also a bit of Kubernetes YAML and some Python for a simple FastAPI service that sends emails based on an HTTP payload, but I’ll focus on the infrastructure.

AWS

The root module for provisioning AWS resources is quite simple. It takes the following variables:

• domain: the domain of an existing Route53 hosted zone (the zone is referenced via a data source)

• gke_oidc_issuer_hostpath: This is where the Google Kubernetes Engine OIDC information and certificates are served from

• k8s_service_account_id: This name of the ServiceAccount resource that’s associated with the k8s pod that sends emails.

main.tf is extremely simple in this case, and points to a module I wrote that handles basic configuration for SES given a Route53 hosted zone.

data "aws_route53_zone" "domain" {
  name = var.domain
}

module "email" {
  source = "../../../modules/aws/ses/"

  mail_domain = "dev.${var.domain}"
  zone_id     = data.aws_route53_zone.domain.id
}

The ima.tf file is a bit more interesting.

The key resource here is aws_iam_openid_connect_provider, which allows AWS IAM to federate with Google Cloud’s OIDC implementation.

resource "aws_iam_openid_connect_provider" "gke_cluster" {
  url            = "https://${var.gke_oidc_issuer_hostpath}"
  client_id_list = ["sts.amazonaws.com"]
}

The remaining part of the file concerns the role that is assumed by the GKE workload. The most important thing here is the trust relationship described by the IAM policy document gke_assume_role. It’s what tells the AWS STS service to grant access to a workload

1. whose identity is provided by the GKE OIDC identity provider

2. and whose identity token refers to a specific kubernetes ServiceAccount

data "aws_iam_policy_document" "send_email" {
  statement {
    actions   = ["ses:SendRawEmail", "ses:SendEmail"]
    resources = ["*"]
  }
}

resource "aws_iam_policy" "send_email" {
  name = "send-emails"

  description = "Allows sending of emails from the development domain"

  policy = data.aws_iam_policy_document.send_email.json
}

data "aws_iam_policy_document" "gke_assume_role" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.gke_cluster.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "${var.gke_oidc_issuer_hostpath}:sub"
      values   = [var.k8s_service_account_id]
    }
  }
}

resource "aws_iam_role" "email_sender" {
  name               = "email-sender"
  assume_role_policy = data.aws_iam_policy_document.gke_assume_role.json
}


resource "aws_iam_role_policy_attachment" "email_sender" {
  role       = aws_iam_role.email_sender.name
  policy_arn = aws_iam_policy.send_email.arn
}

Google Cloud

The root module for the Google Cloud infrastructure contains a lot more variables, many out of scope for this blog post. Here are the relevant ones:

• rm_emailer_aws_role_arn: The ARN that identifies the IAM role granting the GKE workload permission to send emails

• rm_emailer_aws_ses_identity_arn: The ARN that identifies the SES identity used to send emails

• rm_emailer_aws_region: The AWS region in which the SES identity is provisioned

The only terraform code that truly matters here is the workload_identity_config block on the google_container_cluster resource.

  workload_identity_config {
    workload_pool = "${data.google_project.project.project_id}.svc.id.goog"
  }

The three relevant variables are passed into a Helm release for this chart. They’re used in this deployment template:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    {{- include "shortlist-rm-email-notifier.labels" . | nindent 4 }}
  namespace: {{ .Values.namespace }}
  name: {{ include "shortlist-rm-email-notifier.fullname" . }}
spec:
  replicas: {{ .Values.replicas }}
  selector:
    matchLabels:
      {{- include "shortlist-rm-email-notifier.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "shortlist-rm-email-notifier.labels" . | nindent 8 }}
        {{- with .Values.podLabels }}
        {{- toYaml . | nindent 8 }}
        {{- end }}
    spec:
      serviceAccountName: {{ include "shortlist-rm-email-notifier.fullname" . }}
      containers:
      - image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
        name: {{ .Chart.Name }}
        livenessProbe:
          {{- toYaml .Values.livenessProbe | nindent 10 }}
        readinessProbe:
          {{- toYaml .Values.readinessProbe | nindent 10 }}
        resources:
          {{- toYaml .Values.resources | nindent 10 }}
        env:
        - name: AWS_REGION
          value: {{ .Values.aws.region }}
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: '/var/run/secrets/tokens/{{ include "shortlist-rm-email-notifier.fullname" . }}-token'
        - name: AWS_ROLE_ARN
          value: {{ .Values.aws.roleArn }}
        - name: AWS_SES_IDENTITY_ARN
          value: {{ .Values.aws.sesIdentityArn }}
        - name: SOURCE_EMAIL
          value: {{ .Values.emailer.sourceEmail }}
        - name: DESTINATION_EMAIL
          value: {{ .Values.emailer.destinationEmail }}
        volumeMounts:
        - mountPath: /var/run/secrets/tokens
          name: serviceaccount-token-for-aws-sts 
      volumes:
      - name: serviceaccount-token-for-aws-sts 
        projected:
          sources:
          - serviceAccountToken:
              path: '{{ include "shortlist-rm-email-notifier.fullname" . }}-token'
              expirationSeconds: 86400
              audience: sts.amazonaws.com

Most of the template is boilerplate, but here are the bits relevant to this post:

• spec.template.spec.serviceAccountName is set to the same string that’s used in the path to the identity token:

  • spec.template.spec.volumes[0].projected.sources[0].serviceAccountToken.path

  • The AWS_WEB_IDENTITY_TOKEN_FILE environment variable at spec.template.spec.containers[0].env[1]

• The IAM role ARN and SES identity ARN are passed into the container as environment variables to be used by the application code. spec.template.spec.containers[0].env

Conclusion

I should reiterate that this is a hobby project, and I’m not suggesting that this pattern is followed to the letter in production. This article is more of a deep dive into how identity federation can bridge the gap between cloud providers than a tutorial per se.

The biggest flaw is that the Google Cloud and AWS OpenTofu/Terraform root modules depend on each other’s outputs. I got around this by writing the code incrementally, but if you were to start from scratch with my code, I think dummy values would be necessary to get things up and running.