This article has been written as a result of a talk I have given recently within the knowledge exchange (XKE) sessions at Xebia in the Netherlands. The question quite simple:

How can we manage K8s infrastructure and applications using one codebase and high level programming languages?

In the coming paragraphs, we will identify how we can write Infrastructure as Code (IaC) as well as the K8s workload definition for an application that will be deployed on AWS. We will combine the power of AWS CDK and cdk8s in one single codebase to deploy our infrastructure and application.

Declarative vs Imperative approach for IaC

Before we proceed further, let’s first identify the various ways you can write infrastructure as code and the major differences between the imperative and declarative way of working.

Declarative and Imperative approach for programming are two different concepts of developing and delivering code. We will take a closer look at:

Main differences between the two approaches, only focusing on IaC.
Are we really doing it right?
Dive in the specific case of AWS and see the strengths of Terraform over AWS CDK and vice versa.

There are various tools one can use to write IaC. To name a few well known options:

YAML
Crossplane
pulumi
Terraform
AWS CDK

So what is the difference between the declarative and the imperative approach and how do the above options fit in?

Declarative approach

The declarative approach focuses on the “what” of the infrastructure. With a declarative approach, you define the desired end-state of the infrastructure without specifying how to get there. This means that you only need to specify what resources you want to create, configure, or delete. Declarative IaC tools include languages like YAML, JSON, and HCL.

Imperative approach

Focuses on the “how” of the infrastructure. With an imperative approach, you write step-by-step instructions for provisioning and configuring resources. This means that you need to define every step required to create and configure the infrastructure. Imperative IaC tools include scripts written in languages like Bash, PowerShell, or Python.

Terraform VS AWS CDK

Let’s focus for a moment on 2 of the options mentioned above that have most of the markete share in terms of deploying infrastructure in AWS. So what are the differences?

Language and Syntax

Terraform uses its own declarative configuration language (HCL) that is designed to be cloud-agnostic, while AWS CDK allows you to define infrastructure using familiar programming languages like Python, TypeScript, Java, C#, and Go. This means that with AWS CDK, you can leverage the full power and expressiveness of a programming language to define your infrastructure, whereas with Terraform, you use a dedicated configuration language.

Abstraction Level

Terraform provides a higher level of abstraction, allowing you to define infrastructure resources, providers, and modules in a more abstract and cloud-agnostic way. This means that you can use Terraform to manage resources across multiple cloud providers, not just AWS. AWS CDK is specifically designed for provisioning AWS resources and provides a more AWS-centric approach, with AWS-specific constructs, libraries, and APIs that allow you to define AWS resources in a more native way.

Deployment Model

Terraform follows a declarative model, where you define the desired state of your infrastructure and Terraform handles the details of how to achieve that state. Changes to infrastructure are managed by updating the Terraform configuration and applying the changes. AWS CDK, on the other hand, follows an imperative model, where you define the infrastructure using programming code and changes are made by updating the code and redeploying the CDK app. This means that with AWS CDK, you have more flexibility in terms of defining complex logic and dynamic behavior, but it may require additional coding and testing compared to Terraform.

Learning Curve and Familiarity

Terraform’s HCL syntax is designed to be simple and easy to learn, with a configuration-based approach that is familiar to many operations and infrastructure teams. AWS CDK, being a programming-based approach, requires developers to have knowledge and experience with programming languages, which may be an advantage or disadvantage depending on your team’s skill set and preferences.

Is Terraform truly declarative?

The lifecycle meta-argument in Terraform is considered imperative, as it allows you to specify certain lifecycle configuration settings for a resource, which affect how Terraform manages the resource during updates or deletes.

The provisioner meta-argument in Terraform is considered imperative, as it allows you to define actions that are performed on a resource after it has been created or updated, typically used for configuration management or bootstrapping tasks.

Is the AWS CDK truly imperative?

You can actually use the AWS CDK in such a way that the complete codebase will be declarative by avoiding imperative declarations.

Ini the following example, the definition of the S3 bucket can be considered declarative.

Terraform VS AWS CDK (again)

All the above are just a matter of definitions. The choice between Terraform and AWS CDK depends on your specific requirements, preferences, and team’s skill set. Both tools are powerful and widely used in the industry, and they offer different approaches to defining and managing cloud infrastructure as code.

Managing K8s infra and applications on AWS with the AWS CDK and cdk8s

You would typically deploy K8s in the following way: – Create infrastructure using IaC or in any other way. – Define your K8s workload (pods, deployments, statefulsets, etc) using Yaml. – Create your images using Docker. – Configure your cluster with kubectl and deploy.

K8s configuration has always been a declarative paradigm, but then again so has been IaC in its early stages.

Let’s consider the following challenge. We want to: – Deploy K8s infrastructure on AWS using AWS CDK. – Define K8s workload using cdk8s. – Use a single codebase – Use a single step deployment for the infrastructure and the K8s workload on AWS EKS for all the above.

The above use case does not describe a better or preferred way of working with IaC and K8s workload definitions, rather just showing an alternative way of approaching this topic.

Target infrastructure

The target infrastructure we will try to deploy can be seen in the following diagram:

Create you AWS CDK project

We will be creating our AWS CDK project using python. The intention is to create a “template” project that can be reused when someone wants to get starts on AWS using K8s. The same approach can be taken to create similar “templates” in the other programming languages that are supported by the family of the CDK projects.

cdk init app --language=python

More information on getting started with the AWS CDK can be found here.

Define your AWS resources

As per the design shown above, we need to create the following components in AWS:

The network infrastructure
An EKS cluster

Network

For the network, we will eb creating a VPC along with its surrounding resources. We will be using the ec2.Vpc level 2 construct.

Please find the implementation below.

from aws_cdk import (
    NestedStack,
    aws_ec2 as ec2,
)
from constructs import Construct
from helper import config

class NetworkingStack(NestedStack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        conf = config.Config(self.node.try_get_context('environment'))

        self.vpc = ec2.Vpc(self, "k8s-sample-vpc",
            ip_addresses=ec2.IpAddresses.cidr(conf.get('cidr')),
            subnet_configuration=[
                ec2.SubnetConfiguration(
                    name = 'public',
                    subnet_type = ec2.SubnetType.PUBLIC,
                    cidr_mask = 28
                ),
                ec2.SubnetConfiguration(
                    name = 'eks',
                    subnet_type = ec2.SubnetType.PRIVATE_WITH_EGRESS,
                    cidr_mask = 26
                )
            ],
        )

EKS cluster

For the EKS cluster, we will be using the eks.Cluster level 2 construct.

Please find the implementation below.

from aws_cdk import (
    NestedStack,
    aws_eks as eks,
    aws_ec2 as ec2,
    aws_iam as iam
)
from constructs import Construct
from helper import config
from aws_cdk.lambda_layer_kubectl_v25 import KubectlV25Layer

class EKSStack(NestedStack):

    def __init__(self, scope: Construct, construct_id: str, vpc: ec2.IVpc, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        conf = config.Config(self.node.try_get_context('environment'))

        self.cluster = eks.Cluster(
            self, 'k8s-sample-cluster',
            version=eks.KubernetesVersion.V1_25,
            kubectl_layer=KubectlV25Layer(self, "kubectl"),
            alb_controller=eks.AlbControllerOptions(
                version=eks.AlbControllerVersion.V2_4_1
            ),
            default_capacity=0,
            vpc=vpc,
            vpc_subnets=[
                ec2.SubnetSelection(
                    subnet_group_name='eks'
                )
            ]
        )

        self.cluster.add_nodegroup_capacity(
            'eks-nodegroup',
            instance_types=[ec2.InstanceType('t3.large')]
        )

        # importing an existing user
        admin_user = iam.User.from_user_arn(
            self, 'imported_user',
            user_arn='arn:aws:iam::970059968789:user/iam_user' # change me
        )
        self.cluster.aws_auth.add_user_mapping(admin_user, groups=["system:masters"])

The level 2 construct for creating an EKS cluster creates a few important resources for the integration with cdk8s. Namely, 2 lambda functions are deployed together with the cluster:

KubectlHandler is a Lambda function for invoking kubectl commands on the cluster.
ClusterHandler is a Lambda function for interacting with EKS API to manage the cluster lifecycle.

See below for the architectural design of the eks.Cluster level 2 construct.

Integrating cdk8s into the AWS CDK project

In order to start using cdk8s, we need to perform the following changes in the AWS CDK project.

In the requirements.txt, we need to configure the following 2 libraries:
- cdk8s
- cdk8s-plus-25 ¹
Create a folder (e.g. k8s_full_stack_charts) on the root level of the project that will hold you cdk8s charts. Create your chars within that folder as classes that inherit from the cdk8s.Chart class.

Example chart used in this project:

import cdk8s as cdk8s
import cdk8s_plus_25 as kplus

from constructs import Construct

class SampleChart(cdk8s.Chart):
    def __init__(self, scope: Construct, id: str):
        super().__init__(scope, id)

        nginx_deployment = kplus.Deployment(
            self, 'sampleDeployment',
            replicas=1,
            containers=[
                kplus.ContainerProps(
                    image='nginx:mainline-alpine',
                    port=80,
                    security_context=kplus.ContainerSecurityContextProps(
                        ensure_non_root=False,
                        read_only_root_filesystem=False
                    )
                )
            ],
            security_context=kplus.PodSecurityContextProps(
                ensure_non_root=False
            )
        )

        nginx_deployment.expose_via_service(
            service_type=kplus.ServiceType.LOAD_BALANCER
        )

Use the add_cdk8s_chart of the eks.Cluster class to deploy the chart on the EKS Cluster. The deployment will use the KubectlHandler lambda function for executing the relevant kubectl commands.

Example:

from aws_cdk import (
    NestedStack,
    aws_eks as eks
)
import cdk8s as cdk8s
from constructs import Construct

class EKSApplicationStack(NestedStack):

    def __init__(self, scope: Construct, construct_id: str, cluster: eks.ICluster, chart: cdk8s.Chart, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        cluster.add_cdk8s_chart(
            'sample-chart',
            chart,
            ingress_alb=True,
            ingress_alb_scheme=eks.AlbScheme.INTERNET_FACING,
            prune=True
        )

Final thoughts

This way of deploying K8s clusters together with the workload can be really useful in ephemeral clusters. It could also be useful for teams that are proficient with high level programming languages to bundle infrastructure and workloads together using the same high level programming language of their choice.

You can find the complete solution for everything discussed above in GitHub. You can use it as a started template for deploying K8s clusters on AWS together with managing the application in one repository. Next steps would include adding extra templates in the aforementioned repository in the other languages supported by the CDK frameworks family.

Main image by svstudioart on Freepik

[1]: The name/version of this library depends on the K8s version used for the EKS Cluster. In this case we are using K8s version 1.25.

*This article was originally posted here.

Managing K8S Infrastructure and Applications on AWS: The Imperative Way

Table of contents