Managing Multiple Infrastructures with Terraform: tfvars and Workspaces Explained

Terraform is a powerful tool for defining infrastructure as code, allowing you to create resources like servers, databases, and networks. When working on projects with multiple environments (e.g., development, staging, production) or different configurations (e.g., regions or instance sizes), managing these variations efficiently is crucial. Two common methods for handling multiple infrastructures in Terraform are using tfvars files and workspaces. This article explains these methods in simple terms, their pros and cons, whether they’re good to use, and if there’s a better approach, all in a step-by-step format.

---

What Does "Multiple Infrastructures" Mean?

In Terraform, managing multiple infrastructures means deploying similar resources with different configurations. For example:

• Environments: A web application with separate development, staging, and production environments, each with different instance sizes or database settings.

• Regions: Deploying the same application in multiple AWS regions (e.g., us-west-2 and us-east-1).

• Clients: Setting up similar infrastructure for different clients with unique names or settings.

Without a proper strategy, you might duplicate code for each environment, leading to repetition and errors. Terraform’s tfvars and workspaces methods help avoid this by reusing the same code with different inputs.

---

Why Do We Need These Methods?

Managing multiple infrastructures solves several challenges:

1. Avoid Code Duplication: Reusing the same Terraform code for different environments reduces errors and maintenance effort.

2. Consistency: Ensures all environments follow the same structure, differing only in specific settings (e.g., instance type).

3. Scalability: Makes it easier to add new environments or configurations without rewriting code.

4. Flexibility: Allows quick adjustments to settings (e.g., changing a region or resource size) for each environment.

---

Method 1: Using tfvars Files

tfvars files (Terraform variable files) store input variable values for different environments or configurations. They allow you to reuse the same Terraform code by applying different sets of variables.

Step-by-Step: Using tfvars Files

Step 1: Define Variables

Create a variables.tf file to define input variables for your Terraform configuration. For example, for an AWS EC2 instance and S3 bucket:

variable "region" {
  description = "AWS region"
  type        = string
}

variable "instance_type" {
  description = "EC2 instance type"
  type        = string
}

variable "bucket_name" {
  description = "S3 bucket name"
  type        = string
}

Step 2: Write Terraform Code

Create a main.tf file with resources that use these variables:

provider "aws" {
  region = var.region
}

resource "aws_instance" "server" {
  ami           = "ami-12345678"
  instance_type = var.instance_type
}

resource "aws_s3_bucket" "bucket" {
  bucket = var.bucket_name
}

Step 3: Create tfvars Files

Create separate .tfvars files for each environment. For example:

• dev.tfvars:

    region        = "us-west-2"
    instance_type = "t2.micro"
    bucket_name   = "my-app-dev-bucket"
  

• prod.tfvars:

    region        = "us-east-1"
    instance_type = "t3.medium"
    bucket_name   = "my-app-prod-bucket"
  

Step 4: Apply tfvars Files

Run Terraform commands, specifying the tfvars file for the desired environment:

• For development:

    terraform apply -var-file="dev.tfvars"
  

• For production:

    terraform apply -var-file="prod.tfvars"
  

Terraform uses the variables from the specified .tfvars file to create resources with the appropriate settings.

Step 5: Manage State

Terraform creates a single terraform.tfstate file by default, storing the state of all resources. To avoid conflicts between environments:

• Use a backend to store state files separately (e.g., in an S3 bucket).
  Example backend.tf:

    terraform {
      backend "s3" {
        bucket = "my-terraform-state"
        key    = "dev/terraform.tfstate"  # Change key for each environment
        region = "us-west-2"
      }
    }
  

• Update the key (e.g., prod/terraform.tfstate) for each environment to keep state files separate.

Pros of tfvars

• Simple: Easy to set up and understand, especially for small projects.

• Flexible: You can create as many .tfvars files as needed for different configurations.

• Clear Separation: Each environment’s settings are in a dedicated file, making it easy to review.

Cons of tfvars

• State Management: Requires manual state file separation (e.g., different backend keys), which can be error-prone.

• No Isolation: All environments share the same Terraform working directory, increasing the risk of applying the wrong .tfvars file.

• Repetitive Commands: You must specify the correct -var-file flag each time, which can lead to mistakes.

---

Method 2: Using Workspaces

Terraform workspaces allow you to manage multiple environments within the same Terraform configuration by switching between isolated state files. Each workspace has its own state file, but they share the same code.

Step-by-Step: Using Workspaces

Step 1: Define Variables

Use the same variables.tf and main.tf as in the tfvars example. To make workspaces, you can reference the workspace name in your code:

# In main.tf
provider "aws" {
  region = var.region
}

resource "aws_instance" "server" {
  ami           = "ami-12345678"
  instance_type = lookup(var.instance_type, terraform.workspace)
}

resource "aws_s3_bucket" "bucket" {
  bucket = "${var.bucket_name}-${terraform.workspace}"
}

• terraform.workspace returns the current workspace name (e.g., dev or prod).

• lookup selects a value from a map based on the workspace.

Step 2: Define Workspace-Specific Variables

In variables.tf, use maps to define values for each workspace:

variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-west-2"
}

variable "instance_type" {
  description = "EC2 instance type"
  type        = map(string)
  default     = {
    dev  = "t2.micro"
    prod = "t3.medium"
  }
}

variable "bucket_name" {
  description = "Base S3 bucket name"
  type        = string
  default     = "my-app"
}

Step 3: Create Workspaces

Initialize Terraform and create workspaces:

terraform init
terraform workspace new dev
terraform workspace new prod

• Each workspace gets its own state file (e.g., terraform.tfstate.d/dev/terraform.tfstate).

Step 4: Switch and Apply

Switch to a workspace and apply:

• For development:

    terraform workspace select dev
    terraform apply
  

  Creates a t2.micro instance and my-app-dev-bucket in us-west-2.

• For production:

    terraform workspace select prod
    terraform apply
  

  Creates a t3.medium instance and my-app-prod-bucket in us-east-1.

Step 5: Manage State

Workspaces automatically store state files separately in the terraform.tfstate.d directory. For remote state, configure a backend with dynamic keys:

terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "${terraform.workspace}/terraform.tfstate"
    region = "us-west-2"
  }
}

• The ${terraform.workspace} part ensures each workspace uses a unique state file key.

Pros of Workspaces

• Isolated State: Each workspace has its own state file, reducing conflicts between environments.

• Single Directory: All environments share one Terraform directory, simplifying code management.

• Built-in Tooling: Workspaces are a native Terraform feature, with commands like terraform workspace list.

Cons of Workspaces

• Limited Scope: Workspaces only separate state files, not code. You need logic (e.g., lookup) to handle different configurations.

• Complexity: Managing many workspaces or complex configurations can become hard to read.

• Not Ideal for Large Teams: Workspaces don’t enforce strict isolation, and accidental workspace switches can cause issues.

---

Are tfvars and Workspaces Good to Use?

Both methods are good for specific scenarios, but their suitability depends on your project’s size and complexity:

When tfvars Is Good

• Small Projects: If you have a few environments with straightforward differences (e.g., region or instance type), tfvars is simple and effective.

• Clear File-Based Configs: .tfvars files are easy to version control and review in Git.

• No Need for Dynamic Logic: If your configurations are static and don’t require complex workspace-based lookups.

When Workspaces Are Good

• Moderate Projects: Workspaces work well for projects with multiple environments that share the same code but need isolated state files.

• Quick Environment Switching: Useful for developers who frequently switch between environments in one directory.

• Native Integration: No external tooling needed, as workspaces are built into Terraform.

Limitations

• tfvars: Risk of applying the wrong file, manual state management, and lack of code isolation.

• Workspaces: Limited to state separation, not ideal for vastly different infrastructures, and can get messy with many workspaces.

• Both methods struggle with large-scale projects or complex differences (e.g., different providers or architectures per environment).

---

Is There a Better Approach?

For many projects, a directory-based structure with modules is often better than tfvars or workspaces, especially for large or complex infrastructures. This approach uses separate directories for each environment, combined with reusable Terraform modules.

Step-by-Step: Directory-Based Structure with Modules

Step 1: Create a Module

Create a reusable module for your infrastructure (e.g., an EC2 and S3 setup). In modules/ec2_s3/main.tf:

variable "region" {
  type = string
}

variable "instance_type" {
  type = string
}

variable "bucket_name" {
  type = string
}

provider "aws" {
  region = var.region
}

resource "aws_instance" "server" {
  ami           = "ami-12345678"
  instance_type = var.instance_type
}

resource "aws_s3_bucket" "bucket" {
  bucket = var.bucket_name
}

Step 2: Set Up Environment Directories

Create separate directories for each environment:

project/
├── modules/
│   └── ec2_s3/
│       ├── main.tf
│       ├── variables.tf
├── dev/
│   └── main.tf
├── prod/
│   └── main.tf

• In dev/main.tf:

    module "infra" {
      source        = "../modules/ec2_s3"
      region        = "us-west-2"
      instance_type = "t2.micro"
      bucket_name   = "my-app-dev-bucket"
    }
  

• In prod/main.tf:

    module "infra" {
      source        = "../modules/ec2_s3"
      region        = "us-east-1"
      instance_type = "t3.medium"
      bucket_name   = "my-app-prod-bucket"
    }
  

Step 3: Configure State per Environment

Use a backend to store state files separately for each directory:

• In dev/backend.tf:

    terraform {
      backend "s3" {
        bucket = "my-terraform-state"
        key    = "dev/terraform.tfstate"
        region = "us-west-2"
      }
    }
  

• In prod/backend.tf:

    terraform {
      backend "s3" {
        bucket = "my-terraform-state"
        key    = "prod/terraform.tfstate"
        region = "us-east-1"
      }
    }
  

Step 4: Apply per Environment

Navigate to each directory and apply:

cd dev
terraform init
terraform apply

cd ../prod
terraform init
terraform apply

Why Is Directory-Based Structure Better?

• Complete Isolation: Each environment has its own directory, state file, and configuration, reducing the risk of mistakes.

• Scalability: Easily add new environments by creating new directories.

• Modularity: Reusable modules ensure consistent resources across environments.

• Clarity: No need for complex lookup logic or worrying about applying the wrong .tfvars file.

• Team-Friendly: Different teams can work on different directories without conflicts.

When to Use Directory-Based Structure

• Large Projects: When you have many environments or complex configurations.

• Different Architectures: If environments have structural differences (e.g., prod has a load balancer, dev doesn’t).

• Team Collaboration: When multiple developers or teams manage different environments.

Drawbacks

• Setup Overhead: Requires more initial setup (creating directories and module files).

• Code Duplication: If not using modules, you might duplicate code across directories (mitigated by using modules).

---

Comparison Table

---

Recommendations

• Use tfvars for small projects with a few environments and simple differences (e.g., different regions).

• Use Workspaces for moderate projects where you want isolated state files but can manage configurations in one directory.

• Use Directory-Based Structure with Modules for large, complex, or team-based projects where isolation and scalability are critical.

---

Conclusion

Managing multiple infrastructures in Terraform is essential for projects with different environments or configurations. The tfvars method is simple and file-based, ideal for small projects, but lacks isolation and can lead to errors. Workspaces provide state isolation and are great for moderate projects, but they don’t separate code and can get complex. The directory-based structure with modules is the most robust approach for large or complex projects, offering full isolation, scalability, and modularity, though it requires more setup.

Choose the method based on your project’s size, team needs, and complexity. For most real-world scenarios, combining directories with reusable modules provides the best balance of flexibility and maintainability, ensuring your Terraform code remains clean, consistent, and scalable.