Using Terraform to create Lamba Functions in a VPC using AWS Lambda Layers

This post is a follow up to my last article “Traversing Complex and Deeply Nested API Responses with Python”.

In that piece, I demonstrated how we can use the pandas library in python to traverse a deeply nested .json object, extract data from specific dictionary keys, and restructure it as a flattened pandas dataframe. From there, we can use sqlalchemy to open a connection to a MySQL database and insert the data into a table.

Now, I want to go further by invoking a Lambda function that can run this Python code, allowing the data collection, transformation, and loading to be completed within seconds using the power of serverless. To get this working, the Lambda function will need to connect to my VPC so that it can talk to an RDS instance in a private subnet. It will also need internet access so that it can gather the source files from an S3 bucket in my account.

My goal in this post will be to walk through how to use Terraform to configure a VPC with public and private subnets, and create a Lambda function that is ready to go— with its deployment package, VPC access, and connectivity to private resources. I will confirm the Lambda function’s connectivity to the RDS instance with a simple python script that utilizes pymysql.

Prerequisites

This guide assumes that you already have some working knowledge of AWS, the Lambda service, and Terraform. If you want to follow along, you'll need to have an AWS account set up and have Terraform and the AWS CLI installed on your computer.

We will need to set up these prerequisites from the AWS console before writing any .HCL code:

• Set up an AWS user account for Terraform.

• Create an IAM role for the Terraform account (with a trust policy that allows the Terraform user account to assume it).

• Set up the appropriate permission policies for the Terraform IAM role.

Provider and authentication configuration

If you’re not familiar with using Terraform, this is quick overview on how Terraform connects to the cloud and creates our infrastructure. Terraform uses what’s known as a “provider” to connect to orchestrate the environment. Providers are a logical abstraction of an upstream API. They are responsible for understanding API interactions and exposing resources.

In this case, we need to create a providers.tf file or write a provider block in the code. For my demonstration, I am going to create a providers.tf file.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region  = "us-east-2"
  shared_credentials_files = ["path/to/my/.aws/credentials"]
  profile = "terraformuser"

  assume_role {
    role_arn = "arn:aws:iam::###:role/demo_role"
    session_name = "demo_role_session"
    }
}

In this file, I have specified shared_credentials_files to pass the credentials that Terraform will use for authentication as well as specified the IAM role Terraform will use.

After this is done, run the command terraform init to initialize Terraform in your working directory.

VPC configuration

By default, Lambda functions are launched in the Lambda managed VPC which has internet access. In order to set up Lambda functions that can access our resources in a VPC, we need to assign the Lambda function to our VPC. Attaching the function to the VPC means that it will be assigned a private IP address and the Lambda function will not have internet access. This is true even if the function is in a public subnet (a subnet with a route to an internet gateway). If the Lambda resource needs internet access from the VPC, it must use a NAT Gateway or a VPC endpoint for AWS managed services. For this demonstration I will use a NAT Gateway.

Let’s start by writing out the VPC configuration. Create a file in your working directory called main.tf. The following creates a VPC, an internet gateway, a NAT gateway, an elastic IP, a public subnet, and two private subnets (I will also be deploying an RDS instance).

#VPC
resource "aws_vpc" "demo_vpc" {
  cidr_block           = "10.20.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "demo vpc"
  }
}

#subnets
resource "aws_subnet" "demo_public_subnet" {
  vpc_id                  = aws_vpc.demo_vpc.id
  map_public_ip_on_launch = true
  cidr_block              = "10.20.1.0/24"
  availability_zone       = "us-east-2a"

  tags = {
    Name = "public-subnet"
  }
}

resource "aws_subnet" "demo_private_subnet" {
  vpc_id            = aws_vpc.demo_vpc.id
  cidr_block        = "10.20.2.0/24"
  availability_zone = "us-east-2a"

  tags = {
    Name = "private-subnet"
  }
}

resource "aws_subnet" "demo_private_subnet2" {
  vpc_id            = aws_vpc.demo_vpc.id
  cidr_block        = "10.20.3.0/24"
  availability_zone = "us-east-2b"

  tags = {
    Name = "private-subnet2"
  }
}

#igw

resource "aws_internet_gateway" "demo_igw" {
  vpc_id = aws_vpc.demo_vpc.id

  tags = {
    Name = "demo-IGW"
  }

}

#EIP
resource "aws_eip" "elastic_ip" {
  depends_on = [aws_internet_gateway.demo_igw]
  domain     = "vpc"
}

#NAT gw
resource "aws_nat_gateway" "demo_nat_gateway" {
  allocation_id = aws_eip.elastic_ip.id
  subnet_id     = aws_subnet.demo_public_subnet.id

  tags = {
    Name = "demo_nat_gateway"
  }


  # To ensure proper ordering, it is recommended to add an explicit dependency
  # on the Internet Gateway for the VPC.
  depends_on = [aws_internet_gateway.demo_igw]
}

After this, we will need to create the route tables and route table associations to complete the VPC configuration. I won't include my code for this step because it's straightforward if you follow the documentation. I also assume that most readers of an article about Terraform configuration have done this before in some way. Still, it's good to provide a complete overview.

This is a good time to test if our Terraform is configured properly and the authentication we set up is working.

Running the terraform plan will command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.

When terraform plan runs successfully, it will normally create a terraform.tfstate file and output a preview of the changes it will make in the console. This is what it will look like if there are no changes to make:

AWS Lambda Layers

A Lambda function requires you to specify the runtime as well as provide any dependencies that are needed for the code to run. These dependencies are normally delivered with the function code in a .zip file referred to as the package deployment.

As an alternative to this standard approach, you can create a Lambda layer. A Lambda layer is a.zip file archive that contains supplementary code or data. These layers usually contain library dependencies, a custom runtime, or configuration files.

However, just zipping the …/site-packages directory of your Python project and uploading it to Lambda may not let the code run because of compatibility issues. If you are working with Python functions, like I am, it's useful to know that Python packages with compiled code are not always compatible with Lambda runtimes. I found this to be the case with pandas. If you are getting a “Unable to import module" error after invoking a Lambda function and after verifying that the module in question is included in the Lambda deployment .zip, it may be a compatibility issue.

In order to work with these kind packages in Lambda, it is recommended to install the manylinux versions of these packages (.whl) in your local environment and create the deployment package with those. This article provides a step-by-step guide on how to do this: https://repost.aws/knowledge-center/lambda-python-package-compatible

To create my deployment, I first checked that pip version 19.3.0 or higher was running on my machine. Then I used this bash command to install the dependencies to the target folder /python in the current directory.

pip install \        
    --platform manylinux2014_x86_64 \
    --target=python \
    --implementation cp \
    --python-version 3.11 \
    --only-binary=:all: --upgrade \
    pandas pymysql sqlalchemy

I am not including the function code in the layer so I am not concerned with it for the moment. Once all the necessary packages are installed, run the command zip -r deployment.zip python to create the deployment package.

Go to the AWS Console > Lambda > Layer > Create Layer.

Upload the file from your local filesystem to AWS Lambda and create the layer.

Once this is done, copy the arn of the Lambda layer, we will use it later in our main.tf file.

Note: you can create an aws_lambda_layer_version with Terraform. This is a good option especially if your layer’s .zip package is small. For this to work, you must add a depends_on line in the Lambda function resource block or you may get an error when Terraform is creating the function.

AWS has a 50 MB file size limit for uploading the package deployment file via the AWS CLI. Since my package zip is larger, I chose to use the AWS console and prepare the layer in advance, as shown above. However, since Lambda functions can have up to 5 layers, another valid workaround would be to create multiple, smaller deployment packages.

Creating the function’s IAM Role and security group

To ensure the Lambda function has the correct access to private resources, we need to assign it an IAM role and a security group.

The first thing to do is to define the assume role policy. We can do this in Terraform with a data source.

data "aws_iam_policy_document" "assume_role_policy" {
  statement {
    sid    = "assumerole"
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]

  }
}

We create an IAM role in Terraform with an aws_iam_role resource. aws_iam_role resources require an assume_role_policy which we just created with the data source. We reference this policy document.

resource "aws_iam_role" "iam_for_demofunction" {
  name               = "iam_for_demofunction"
  assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json

}

Next, we can attach the AWS managed policies needed to Lambda to access resources in a VPC using aws_iam_role_policy_attachment.

resource "aws_iam_role_policy_attachment" "demofunction_vpcexecution_policy_attach" {
  role       = aws_iam_role.iam_for_demofunction.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

resource "aws_iam_role_policy_attachment" "demofunction_basicexecution_policy" {
  role       = aws_iam_role.iam_for_demofunction.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Because my Lambda function needs to be able to access the project’s S3 bucket and connect to an RDS instance, I opted to include these additional policies separately and attach them in the same way.

data "aws_iam_policy_document" "demofunction_inline_policy_document" {

  statement {
    sid    = "allRDS"
    effect = "Allow"

    actions   = ["rds:*"]
    resources = ["*"]
  }

  statement {
    sid    = "allactionsS3projectbucket"
    effect = "Allow"

    actions = ["s3:*"]
    resources = ["arn:aws:s3:::bucket_name",
      "arn:aws:s3:::bucket_name/*"
    ]
  }

}

resource "aws_iam_policy" "demofunction_inline_role_policies" {
  name        = "demofunction_inline_role_policies"
  description = "Adding additional inline role policies for demofunction"
  policy      = data.aws_iam_policy_document.demofunction_inline_policy_document.json
}

resource "aws_iam_role_policy_attachment" "demofunction_inline_role_policies" {
  role       = aws_iam_role.iam_for_demofunction.name
  policy_arn = aws_iam_policy.demofunction_inline_role_policies.arn
}

Next, we can create a security group for the Lambda function. I will also be creating a security group for my RDS instance (which is the private resource my Lambda function connects to).

resource "aws_security_group" "demo_lambdasg" {
  name        = "demo_lambdasg"
  description = "security group for lambda"
  vpc_id      = aws_vpc.demo_vpc.id
}

resource "aws_security_group" "demo_dbsg" {
  name        = "demo_dbsg"
  description = "security group for database"
  vpc_id      = aws_vpc.demo_vpc.id

}

Finally, define the egress and ingress rules that allow the Lambda function to connect to the database.

resource "aws_vpc_security_group_ingress_rule" "allowtraffictodbfromlambda" {
  referenced_security_group_id = aws_security_group.demo_lambdasg.id
  from_port                    = 3306
  to_port                      = 3306
  ip_protocol                  = "tcp"
  security_group_id            = aws_security_group.demo_dbsg.id
}

resource "aws_vpc_security_group_egress_rule" "allowoutboundtrafficfromlambda" {
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = -1
  security_group_id = aws_security_group.demo_lambdasg.id
}

Creating the function

With the functions layer, IAM role, and security group created, we can write the resource block for the function itself. Here, we have a Lambda deployed with a python 3.11 runtime and a 15 second timeout. You can create a zip file of the function’s code and enter the path using the filename argument. This could also be s3_bucket or image_uri. You must select one of these.

resource "aws_lambda_function" "demofunction" {
  filename      = "/path/to/function/index.zip"
  function_name = "demofunction"
  layers        = ["arn:aws:lambda:us-east-2:######:layer:layer_name:#"]
  role          = aws_iam_role.iam_for_demofunction.arn
  handler       = "index.lambda_handler"

  runtime = "python3.11"
  timeout = 15

  vpc_config {
    subnet_ids         = [aws_subnet.demo_private_subnet.id]
    security_group_ids = [aws_security_group.demo_lambdasg.id]
  }

   environment {
       variables = {
         DBUSER     = ###
         DBPASSWORD = ###
         DBHOST     = your-instance-name.your-region.rds.amazonaws.com
       }
     }
}

For layers, paste the arn string for the corresponding layer that was collected in the previous section.

Reference the subnet_id for the private subnet and the security group to complete the VPC configuration.

I’ve included environment variables so that Lambda can login to the RDS instance in my VPC. For better security, you could use the AWS SSM parameter store to securely access the DBUSER and DBPASSWORD variable since these values are sensitive. In Terraform, this can be referenced with a data source. If you also used Terraform to create the RDS instance, you can use the address attribute of the database resource to reference its DNS name.

Alternatively, this looks like:

environment {
       variables = {
         DBUSER     = data.aws_ssm_parameter.DBUSER.value
         DBPASSWORD = data.aws_ssm_parameter.DBPASSWORD.value
         DBHOST     = aws_db_instance.demo_database.address
       }
     }

Save the file.

Apply

We can run terraform plan again to see what changes Terraform will make to our state. This time it shows all of the planned changes dictated by the main.tf file.

Now, run terraform apply -auto-approve and wait for Terraform to provision the resources.

I will run a quick test to ensure that this is working as intended.

My Lambda function will run this simple script to connect to the RDS instance and create a table named “users”.

import pymysql
import os

def lambda_handler(event, context):

    connection = pymysql.connect(
        host=os.getenv("DBHOST"),         
        user=os.getenv("DBUSER"),     
        password=os.getenv("DBPASSWORD"), 
        database="test", 
        charset="utf8mb4",        
    )

    try:
        # Open a cursor to perform database operations
        with connection.cursor() as cursor:
            # Write an SQL statement to create a table if it doesn't exist
            create_table_query = """
            CREATE TABLE IF NOT EXISTS users (
                id INT AUTO_INCREMENT PRIMARY KEY,
                name VARCHAR(100) NOT NULL,
                email VARCHAR(100),
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
            """
            # Execute the SQL statement
            cursor.execute(create_table_query)

        # Commit the transaction to apply changes
        connection.commit()
        print("Table 'users' created successfully!")
        return 200

    except pymysql.MySQLError as error:
        print("Error while connecting to MySQL or executing the query:", error)
        return 500

    finally:
        connection.close()

Calling the function from the AWS console returns a 200 "OK," indicating that the connection was successful and the table was created.

The end.