Crossplane in EKS
Srujan ReddyCrossplane is a CNCF project which can be used to manage AWS resources lifecycle and access them from the application pods. This allows developers to quickly use S3 buckets, SNS or SES resources for their applications and delete them along with the app. Crossplane is a great way to manage AWS resources from a common Kubernetes control-plane instead of trying to figure out what resources you have up etc. Read more here.
Installing Crossplane on a cluster
I have provisioned the EKS cluster using terraform module and enabled OIDC.
Outputs:
cluster_endpoint = "https://C3512060B49A940D946C54B5FF5F4535.gr7.us-east-2.eks.amazonaws.com"
cluster_name = "education-eks-uWa4XrtJ"
cluster_security_group_id = "sg-012181f8810cb1e5b"
region = "us-east-2"
OpenID Connect (OIDC) is an authentication layer built on the OAuth protocol, extending OAuth's authorization abilities to include authentication. OIDC issues an "ID token," a JSON Web Token (JWT) containing user information, akin to an ID card. These ID tokens are proof of authentication, unlike access tokens which can be acquired multiple ways and are not directly tied to user authentication. OIDC standardizes scopes, defining a protected resource that contains common user information. The protocol issues an ID token alongside an access token when the 'openid' scope is requested, easing the process of accessing basic user data.
Ref: https://goteleport.com/blog/how-oidc-authentication-works/
You can install crossplane on the cluster using single helm command
➜ learn-terraform-provision-eks-cluster git:(main) ✗ helm install crossplane \
crossplane-stable/crossplane \
--namespace crossplane-system \
--create-namespace
NAME: crossplane
LAST DEPLOYED: Sat Feb 24 14:25:58 2024
NAMESPACE: crossplane-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Release: crossplane
Chart Name: crossplane
Chart Description: Crossplane is an open source Kubernetes add-on that enables platform teams to assemble infrastructure from multiple vendors, and expose higher level self-service APIs for application teams to consume.
Chart Version: 1.15.0
Chart Application Version: 1.15.0
Verify the crossplane running by checking the pods in crossplane-system namespace
➜ ~ k get pods -n crossplane-system
NAME READY STATUS RESTARTS AGE
crossplane-5b87fcbc66-h8d6g 1/1 Running 0 117m
crossplane-rbac-manager-b884bb468-2n9cp 1/1 Running 0 117m
provider-aws-iam-28cb496c46b8-65c866b566-lp2pf 1/1 Running 0 116m
provider-aws-s3-13463c1ac7d2-5f85d45dfd-6h8ww 1/1 Running 0 116m
upbound-provider-family-aws-3756efeec089-78df6446f6-kdcxc 1/1 Running 0 116m
You may not have these number of pods but you must have the first 2. Now you have to add a crossplane Provider to your cluster to create/manage your resources
Provider is a Crossplane CRD which installs all the required CRDs for that resource.
Here I have installed S3 and IAM providers using definition file
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws-s3
spec:
package: xpkg.upbound.io/upbound/provider-aws-s3:v1.1.0
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-aws-iam
spec:
package: xpkg.upbound.io/upbound/provider-aws-iam:v1.1.0
These provides installs CRDs to create AWS resources directly inside the cluster.
➜ crossplane k get crds |grep s3
bucketaccelerateconfigurations.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketacls.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketanalyticsconfigurations.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketcorsconfigurations.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketintelligenttieringconfigurations.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketinventories.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketlifecycleconfigurations.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketloggings.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketmetrics.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketnotifications.s3.aws.upbound.io 2024-02-24T20:27:16Z
bucketobjectlockconfigurations.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketobjects.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketownershipcontrols.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketpolicies.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketpublicaccessblocks.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketreplicationconfigurations.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketrequestpaymentconfigurations.s3.aws.upbound.io 2024-02-24T20:27:17Z
buckets.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketserversideencryptionconfigurations.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketversionings.s3.aws.upbound.io 2024-02-24T20:27:17Z
bucketwebsiteconfigurations.s3.aws.upbound.io 2024-02-24T20:27:17Z
objectcopies.s3.aws.upbound.io 2024-02-24T20:27:17Z
objects.s3.aws.upbound.io 2024-02-24T20:27:17Z
You need to create secret which contains your cloud credentials
[default]
aws_access_key_id =
aws_secret_access_key =
crossplane k create secret generic aws-secret -n crossplane-system --from-file=creds=./aws-credentials.txt
Now you need to create providerConfig, which contains your cloud secrets, whithout the providerConfig you cannot create resources.
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: Secret
secretRef:
namespace: crossplane-system
name: aws-secret
key: creds
now you can create aws resources like bucket with below definition file.
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
generateName: crossplane-bucket-
spec:
forProvider:
region: us-east-2
providerConfigRef:
name: default
➜ crossplane k get buckets
NAME READY SYNCED EXTERNAL-NAME AGE
crossplane-bucket-jvx8m True True crossplane-bucket-jvx8m 132m
Using IRSA to access AWS resources in pod.
IRSA also called as IAM resources for ServiceAccounts which links the serviceaccount to iam roles. Using this you can have a role for pods and using that role you can use AWS resources like S3, SNS, SES etc.
To demonstrate the same I have created role and attached S3 full access managed policy to it
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*",
"s3-object-lambda:*"
],
"Resource": "*"
}
]
}
And in trust policy, you need to include your OIDC config as well.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::17483678901:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/D510BCG4F7E27AM04AZNS95G6914V4A0"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.ap-south-1.amazonaws.com/id/D510BCG4F7E27AM04AZNS95G6914V4A0:aud": "sts.amazonaws.com",
"oidc.eks.ap-south-1.amazonaws.com/id/D510BCG4F7E27AM04AZNS95G6914V4A0:sub": "system:serviceaccount:demo-s3:demo-sa"
}
}
}
]
}
Then you need to create ServiceAccount and annotate the SA with role
➜ crossplane k annotate sa bucket-sa eks.amazonaws.com/role-arn=arn:aws:iam::135227014767:role/bucket-role-for-crossplane
serviceaccount/bucket-sa annotated
So whenever this SA is attached to the pod, JWT token will be mounted in the pod which assumes the assigned IAM role
I have created a deployment to show the same
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2024-02-24T21:16:50Z"
generation: 1
labels:
app: aws-cli-deploy
name: aws-cli-deploy
namespace: default
resourceVersion: "41708"
uid: 68a358b1-74c1-45a6-bc50-d547968cbe9a
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: aws-cli-deploy
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: aws-cli-deploy
spec:
containers:
- args:
- "1500"
command:
- sleep
image: amazon/aws-cli
imagePullPolicy: Always
name: aws-cli
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: bucket-sa
serviceAccountName: bucket-sa
terminationGracePeriodSeconds: 30
Once you attach the ServiceAccount to pod, EKS will automatically mount JWT token in the pod at this path
Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: us-east-2
AWS_REGION: us-east-2
AWS_ROLE_ARN: arn:aws:iam::135227014767:role/bucket-role-for-crossplane
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lpz8m (ro)
With the role attached, I can now list the S3 resources
bash-4.2# aws s3 ls
2023-10-07 20:13:44 cf-templates-1bjk9y0pvs5y3-us-east-2
2024-02-24 20:29:36 crossplane-bucket-jvx8m
Conclusion
Crossplane is a great alternative for terraform to create, manage resources in Kubernetes paradigm
Subscribe to my newsletter
Read articles from Srujan Reddy directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
Srujan Reddy
Srujan Reddy
I am a Kubernetes Engineer passionate about leveraging Cloud Native ecosystem to run secure, efficient and effective workloads.