How I built a Serverless Movies API on AWS Using Terraform and Go

Over the past month, I’ve been exploring two powerful tools in the AWS ecosystem: Terraform and the Serverless Framework. To truly grasp its power and intricacies, I decided to put my learning into practice by building a real-world project. I built a simple Movies API that runs entirely on a serverless architecture using AWS services.

This article will walk you through the technologies I used, the challenges I encountered, and the solutions I discovered along the way.

Tech Stack

Here's a breakdown of the technologies that I used in this simple project:

• Terraform for IaC (Infrastructure as Code): Terraform is a powerful tool for managing infrastructure as code, allowing you to define and provision your infrastructure in a declarative manner. Terraform lets you describe your infrastructure in code, enabling automation, version control, and repeatability.

• AWS for Infrastructure: I chose AWS as the cloud provider for this project. Having used AWS for over a year, I'm already familiar with its services and ecosystem.

• Go for Backend APIs: While I have experience with Python, I decided to use Go for the backend APIs. This was a deliberate choice to explore and practice Go, adding a layer of challenge and learning to the project.

• Serverless on AWS: To simplify infrastructure management and optimize costs, I opted for a serverless architecture on AWS.

Why Terraform? My Journey from Manual Provisioning

Like many who start with cloud services, I initially created AWS resources manually through the console. While this provides a good understanding of the individual services, it quickly becomes cumbersome, error-prone, and difficult to manage at scale. This realization led me to explore the world of Infrastructure as Code (IaC).

I initially considered AWS CloudFormation. CloudFormation is AWS's native IaC service, defining infrastructure as JSON or YAML templates. However, I was drawn to Terraform for several reasons:

• Multi-Cloud Support: Terraform's ability to manage infrastructure across multiple cloud providers (AWS, Azure, GCP, etc.) offers greater flexibility and avoids vendor lock-in.

• Human-Readable Configuration: Terraform uses a declarative configuration language (HCL) that is generally considered more human-readable and easier to understand than CloudFormation's JSON/YAML.

• State Management: Terraform's robust state management keeps track of your infrastructure, allowing for accurate planning and applying of changes.

This led me to dive into Terraform. It looked promising, and after exploring its concepts, I was eager to get started.

Getting Started with Terraform: Learning the Developer Way

As a developer, I know the best way to learn a new technology is by getting my hands dirty with the documentation. I started to go through the official Terraform docs and followed several tutorials. This gave me a solid understanding of core concepts like providers, resources, and state.

You can also start your own Terraform journey here:

• Install Terraform: Terraform Install

• Terraform Documentation: Terraform Docs

AWS Serverless: Efficiency and Cost-Effectiveness

The decision to go serverless on AWS was driven by a desire to focus on the application logic rather than the underlying infrastructure. I didn't want to spend time managing EC2 instances or setting up and maintaining an RDS database and also wanted to go with NoSQL

Serverless offered several advantages:

• Simplified Operations: AWS handles the underlying infrastructure, allowing me to concentrate on building the API.

• Scalability: Serverless services like Lambda and DynamoDB automatically scale based on demand.

• Cost Optimization: With pay-as-you-go pricing, I only incur costs when my API is actively being used, which was particularly beneficial as I was frequently creating and destroying the infrastructure during development.

You can explore the vast world of AWS services here: AWS

AWS Services in Action

Here are the specific AWS services I leveraged to build the movies API:

• API Gateway: I used API Gateway with a proxy setup to handle incoming HTTP requests and route them to the appropriate backend logic without needing to define every API resource individually.

• Lambda: Lambda functions hosted the Go API code and all the business logic.

• DynamoDB: DynamoDB served as the NoSQL database to store movie details.

• S3: Used S3 for storing movie cover images

• Bonus: Bedrock: I even explored a small aspect of Generative AI by using Amazon Bedrock to generate movie summaries.

The ease with which I could create and destroy the entire infrastructure using Terraform was a game-changer. In scenarios with limited time, I could quickly spin up the necessary resources, build and test the API, and then destroy everything down, saving time compared to manual provisioning.

For detailed information on the AWS Terraform provider, check out the registry: AWS Terraform Registry

Why Go for APIs? A Journey of Exploration

While I'm comfortable with Python, I saw this project as an opportunity to further explore Go. Using Go for the backend APIs presented a more significant challenge and provided valuable hands-on experience with the language.

You can learn more about Go here:

• Go Documentation: Go Docs

• Go by Example: Go By Example

To integrate with AWS services like DynamoDB, Bedrock, and S3, I utilized the AWS SDK for Go.

• AWS SDK for Go V2 Developer Guide: AWS SDK for Go

Fixing My Issues: What I Learned

Building this project wasn't without its mistakes. Here are some of the issues I faced and how I fixed them:

Issue 1: Handling multipart/form-data in API Gateway

My POST and PUT API needed to accept movie images as input, which naturally led to using multipart/form-data. While the API was functioning, I struggled to extract the data from the form data payload.

After some online research and conversations with ChatGPT, I discovered that I needed to explicitly configure API Gateway to handle binary media types by adding binary_media_types = ["multipart/form-data"] to the API Gateway resource in my Terraform configuration.

However, even after applying this change with terraform apply, the API still wasn't working as expected. Further looking revealed that the API deployment wasn't being triggered by the changes to the binary_media_types.

The aws_api_gateway_deployment resource has a triggers argument that determines when a new deployment should occur. Since simply changing binary_media_types doesn't modify any of the tracked resources in the default triggers hash, a new deployment wasn't started.

To resolve this, I had to explicitly include the binary_media_types attribute of the aws_api_gateway_rest_api resource in the triggers argument of the aws_api_gateway_deployment resource:

# main.tf

resource "aws_api_gateway_deployment" "movies_api_deployment" {
  rest_api_id = aws_api_gateway_rest_api.movies_api_gateway.id

  triggers = {
    redeployment = sha1(jsonencode([
      aws_api_gateway_resource.movies_api_resource.id,
      aws_api_gateway_resource.movies_proxy_resource.id,
      aws_api_gateway_method.movies_any_method.id,
      aws_api_gateway_integration.movies_lambda_integration.id,
      aws_api_gateway_rest_api.movies_api_gateway.binary_media_types # Added this line
    ]))
  }

  lifecycle {
    create_before_destroy = true
  }
}

With this addition, any change to the binary_media_types would now trigger a redeployment, and the API started correctly processing the multipart/form-data.

Issue 2: S3 Bucket Policy Conflicts During Initial Deployment

Another challenge was with the S3 bucket policy. During the initial terraform apply when creating the entire infrastructure from scratch, the process would often fail at the aws_s3_bucket_policy resource. However, running terraform apply a second time would succeed without any issues.

This failure pointed towards a dependency issue, likely related to Terraform's state management. The aws_s3_bucket_policy resource depends on the S3 bucket and the public access block configuration (aws_s3_bucket_public_access_block).

Terraform state keeps track of the resources it manages. During the initial run, the order in which resources are created might lead to the bucket policy being applied before the public access block is fully established in the AWS backend, causing a temporary conflict. On subsequent runs, the state is already populated, and Terraform can manage the dependencies more effectively.

To address this, I used the depends_on argument in the aws_s3_bucket_policy resource to ensure it waits for the aws_s3_bucket_public_access_block to be applied first:

# main.tf

resource "aws_s3_bucket_policy" "allow_get_images_policy" {
  bucket = aws_s3_bucket.movies_rest_api_bucket.id
  policy = data.aws_iam_policy_document.allow_get_s3_images_policy.json

  depends_on = [aws_s3_bucket_public_access_block.movies_rest_api_bucket_public_access]
}

This explicit dependency ensured that the bucket policy was only applied after the public access block was in place, resolving the failure during the initial infrastructure creation.

I know I've been chatting about this and that, and you're just eager to see the code. No worries, I've got you covered! You can find the code on my GitHub.

Explore the Code

You can find all the code resources and Terraform scripts for this simple project on my GitHub repository: Github Repo

Feel free to clone the repository, explore the code, and refer to the README file for setup and getting started instructions.

Conclusion

This project has been an invaluable learning experience, solidifying my understanding of Terraform and AWS serverless services.

I'm also open to suggestions! What do you think about what can be done better. Let me know in the comments below!

Looking forward to next time! Peace✌️