Terraform Associate: Refactor Monolithic Terraform Configuration

As your infrastructure grows, managing a single, monolithic Terraform configuration can quickly become overwhelming. Refactoring your configuration into smaller, modular components not only improves readability and maintainability but also increases reusability across projects and environments. In this guide, we’ll explore the process of breaking down a large configuration into focused modules, along with best practices for a scalable infrastructure.

1. Recognizing the Need to Refactor

When your Terraform configuration starts handling multiple responsibilities—networking, compute, storage, databases, and IAM—it becomes difficult to manage and troubleshoot. Look for these common indicators:

• Networking: VPCs, subnets, routing tables, and security groups are all lumped together.

• Compute: EC2 instances, autoscaling groups, and load balancers share the same configuration.

• Storage & Databases: S3 buckets, EBS volumes, RDS, and DynamoDB resources are defined in one file.

• IAM: Roles, policies, and permissions intermixed with other resources.

By isolating these sections into individual modules, you achieve a clear separation of concerns.

2. Organizing Your Project Structure

A well-organized directory structure lays the foundation for modularization. A suggested structure might look like this:

terraform-project/
│── main.tf              # Root configuration that calls modules
│── modules/
│   ├── vpc/             # VPC module
│   ├── ec2-instance/    # EC2 instance module
│   ├── rds/             # RDS module
│   └── s3/              # S3 module
│── variables.tf         # Global input variables
│── outputs.tf           # Outputs for the entire project

This layout keeps your code tidy and makes it easier for teams to collaborate and update individual modules without affecting the entire infrastructure.

3. Extracting and Refactoring Configuration into Modules

Example: Refactoring VPC Resources

Suppose your monolithic main.tf contains a complete VPC configuration like this:

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "subnet_a" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "10.0.1.0/24"
}

resource "aws_security_group" "sg" {
  vpc_id = aws_vpc.main.id
  name   = "allow_all"
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Refactoring Steps:

1. Create a VPC Module Directory:
   Create a new directory modules/vpc/ and move your VPC resources there.

2. Define the Module (modules/vpc/main.tf):

    resource "aws_vpc" "main" {
      cidr_block = var.cidr_block
    }
   
    resource "aws_subnet" "subnet_a" {
      vpc_id     = aws_vpc.main.id
      cidr_block = var.subnet_cidr
    }
   
    resource "aws_security_group" "sg" {
      vpc_id = aws_vpc.main.id
      name   = "allow_all"
      ingress {
        from_port   = 80
        to_port     = 80
        protocol    = "tcp"
        cidr_blocks = var.cidr_blocks
      }
    }
   

3. Define Module Variables (modules/vpc/variables.tf):

    variable "cidr_block" {
      type        = string
      description = "The CIDR block for the VPC"
    }
   
    variable "subnet_cidr" {
      type        = string
      description = "The CIDR block for the subnet"
    }
   
    variable "cidr_blocks" {
      type        = list(string)
      description = "List of CIDR blocks for ingress"
    }
   

4. Use the Module in Your Root Configuration (main.tf):

    module "vpc" {
      source        = "./modules/vpc"
      cidr_block    = "10.0.0.0/16"
      subnet_cidr   = "10.0.1.0/24"
      cidr_blocks   = ["0.0.0.0/0"]
    }
   

By extracting the VPC configuration into its own module, you simplify the main configuration and make the VPC setup reusable in other projects.

4. Breaking Down Other Resources

Refactor additional components similarly:

• EC2 Instances:
  Create a module in modules/ec2-instance/ that includes resource definitions for EC2 instances, input variables for AMI and instance type, and outputs for instance IDs.

    resource "aws_instance" "web" {
      ami           = var.ami
      instance_type = var.instance_type
  
      tags = {
        Name = var.instance_name
      }
    }
  

• S3 Buckets & RDS:
  Develop separate modules for S3 buckets and RDS configurations, each with its own set of variables and outputs. This approach not only organizes your code but also allows you to reuse these modules across different environments (development, production, etc.).

5. Managing Variables and Outputs

To centralize configuration and enhance modularity, create global variables and outputs at the root level.

Global Variables (variables.tf):

variable "ami" {
  description = "AMI ID for EC2 instance"
  type        = string
}

variable "instance_type" {
  description = "EC2 instance type"
  type        = string
}

Global Outputs (outputs.tf):

output "ec2_instance_id" {
  value = module.ec2_instance.instance_id
}

output "vpc_id" {
  value = module.vpc.vpc_id
}

This structure enables different modules to interact seamlessly and provides a clear overview of the entire infrastructure’s outputs.

6. Reusing Modules Across Projects

Once you have modularized your configuration, you can easily reuse these modules in other projects. Simply reference the module by its path or, if it’s hosted in a version control system, by its repository URL:

module "vpc" {
  source        = "/path/to/vpc/module"
  cidr_block    = "10.0.0.0/16"
  subnet_cidr   = "10.0.1.0/24"
  cidr_blocks   = ["0.0.0.0/0"]
}

This portability is one of the key advantages of a modular infrastructure as code approach.

7. Benefits of Modularization

• Improved Organization: A modular structure clearly separates different parts of your infrastructure.

• Enhanced Reusability: Modules can be reused in multiple environments, reducing duplicate code.

• Simplified Maintenance: Update a module in one location and have the changes propagate across all projects that use it.

• Better Collaboration: Teams can work on different modules independently, reducing the risk of conflicts.

8. Best Practices for Modular Terraform Configurations

• Keep Modules Focused: Each module should have a single responsibility, such as provisioning a VPC or launching an EC2 instance.

• Descriptive Naming and Documentation: Use clear variable names and document inputs and outputs to help others understand the module’s purpose.

• Version Control: Manage modules with version control to ensure updates do not break your infrastructure.

• Test Independently: Validate and test each module on its own before integrating it into a larger configuration.

• Avoid Hardcoding: Use variables to parameterize your configurations, making them flexible and adaptable.

Conclusion

Refactoring a monolithic Terraform configuration into smaller, focused modules is a powerful strategy to improve your infrastructure's maintainability, readability, and reusability. By following the steps outlined in this guide, you can transform your sprawling configuration into a well-organized, modular architecture that scales with your infrastructure needs.

Adopt modularization best practices to streamline your workflows, reduce errors, and enable your team to collaborate more effectively. Embrace the journey towards a more maintainable infrastructure, and enjoy the benefits of a cleaner, more efficient Terraform codebase. Happy refactoring!

Reference

1. Terraform Modules: Official Documentation
   A comprehensive resource that explains the fundamentals of modules, how to structure them, and how to incorporate them into your Terraform configuration.

2. Terraform Module Best Practices (Gruntwork)
   Gruntwork outlines practical patterns and pitfalls for building scalable, production-grade modules.

3. HashiCorp Learn - Build and Use a Local Module
   A hands-on tutorial walking you through the process of turning a monolithic Terraform configuration into smaller, reusable components.

4. Terraform Style Guide (Anton Babenko)
   A widely respected style guide covering everything from naming conventions and structure to module design tips.

5. Yevgeniy Brikman: How to create reusable Terraform modules
   A deep dive from one of the co-founders of Gruntwork explaining why modules matter, how to design them well, and how to avoid common anti-patterns.