Introduction

A long awaited request from Terraform users has been implemented in OpenTofu v1.9.x and that is for_each at the provider level. Users of Terraform either used Terragrunt to manage this, or had a lot of code duplication in their configurations. Using multiple providers for AWS helps practitioners deploy configurations into multiple AWS Regions. You might have an application that serves customers in Ireland, the UK and Germany. As a result, you might choose to deploy to eu-west-1 (Ireland), eu-west-2 (London) and eu-central-1 (Frankfurt) for performance or even legal/regulatory requirements.

We’re going to compare the differences in code doing this with Terraform and then with OpenTofu v1.9.x! Specifically the “rc2” release of OpenTofu v1.9.0.

Using Terraform

Given the scenario above, lets deploy a basic VPC setup to those three regions using Terraform:

Terraform v1.10.3
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.82.2

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0.0, < 6.0.0"
    }
  }
}

provider "aws" {
  alias  = "eu_west_1"
  region = "eu-west-1"
}

provider "aws" {
  alias  = "eu_west_2"
  region = "eu-west-2"
}

provider "aws" {
  alias  = "eu_central_1"
  region = "eu-central-1"
}

module "eu_west_1_vpc" {
  source = "git::https://github.com/kieran-lowe/reusable-modules.git//modules/super-duper-basic-vpc?ref=vpc-0.1.0"

  providers = {
    aws = aws.eu_west_1
  }

  vpc_cidr = "10.0.0.0/16"

  subnets = {
    private-1 = {
      cidr = "10.0.0.0/24"
      az   = "eu-west-1a"
    }
    private-2 = {
      cidr = "10.0.1.0/24"
      az   = "eu-west-1b"
    }
  }
}

module "eu_west_2_vpc" {
  source = "git::https://github.com/kieran-lowe/reusable-modules.git//modules/super-duper-basic-vpc?ref=vpc-0.1.0"

  providers = {
    aws = aws.eu_west_2
  }

  vpc_cidr = "10.0.0.0/16"

  subnets = {
    private-1 = {
      cidr = "10.0.0.0/24"
      az   = "eu-west-2a"
    }
    private-2 = {
      cidr = "10.0.1.0/24"
      az   = "eu-west-2b"
    }
  }
}

module "eu_central_1_vpc" {
  source = "git::https://github.com/kieran-lowe/reusable-modules.git//modules/super-duper-basic-vpc?ref=vpc-0.1.0"

  providers = {
    aws = aws.eu_central_1
  }

  vpc_cidr = "10.0.0.0/16"

  subnets = {
    private-1 = {
      cidr = "10.0.0.0/24"
      az   = "eu-central-1a"
    }
    private-2 = {
      cidr = "10.0.1.0/24"
      az   = "eu-central-1b"
    }
  }
}

If we run a terraform plan we get:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.eu_central_1_vpc.aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-central-1a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.eu_central_1_vpc.aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-central-1b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.eu_central_1_vpc.aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "example-vpc"
        }
    }

  # module.eu_west_1_vpc.aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-1a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.eu_west_1_vpc.aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-1b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.eu_west_1_vpc.aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "example-vpc"
        }
    }

  # module.eu_west_2_vpc.aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-2a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.eu_west_2_vpc.aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-2b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.eu_west_2_vpc.aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "Terraform"
          + "Name"         = "example-vpc"
        }
    }

Plan: 9 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: tf.plan

To perform exactly these actions, run the following command to apply:
    terraform apply "tf.plan"

We can see there is a lot of duplication in this configuration. The provider and module configurations are the same with the exception of the alias for each AWS provider and module attributes.

Using OpenTofu

Now let’s define our configuration using OpenTofu!

OpenTofu v1.9.0-rc2
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.82.2

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0.0, < 6.0.0"
    }
  }
}

variable "regions" {
  type    = set(string)
  default = ["eu-west-1", "eu-west-2", "eu-central-1"]
}

variable "disabled_regions" {
  type    = set(string)
  default = []
}

provider "aws" {
  for_each = var.regions

  alias  = "by_region"
  region = each.value
}

module "network" {
  source = "git::https://github.com/kieran-lowe/reusable-modules.git//modules/super-duper-basic-vpc?ref=vpc-0.1.0"

  for_each = setsubtract(var.regions, var.disabled_regions)

  providers = {
    aws = aws.by_region[each.key]
  }

  vpc_cidr = "10.0.0.0/16"

  subnets = {
    private-1 = {
      cidr = "10.0.0.0/24"
      az   = "${each.key}a"
    }
    private-2 = {
      cidr = "10.0.1.0/24"
      az   = "${each.key}b"
    }
  }
}

Now we run a tofu plan and we get:

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # module.network["eu-central-1"].aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-central-1a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-central-1"].aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-central-1b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-central-1"].aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
    }

  # module.network["eu-west-1"].aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-1a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-west-1"].aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-1b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-west-1"].aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
    }

  # module.network["eu-west-2"].aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-2a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-west-2"].aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-2b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-west-2"].aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
    }

Plan: 9 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: t.plan

To perform exactly these actions, run the following command to apply:
    tofu apply "t.plan"

Here we can see we have done it with much less code!

Preventing Unmanageable Resources

I mentioned earlier the rationale behind the setsubtract() function in our for_each configuration at the module level. In short, OpenTofu can only manage resources if the provider they belong to exists in the configuration. This section is going to go into detail on why we do this and what errors we will get if we don’t use it. The setsubtract() function returns all items from the first set that does not exist in the second set, in a new set. Let’s validate this using tofu console:

> setsubtract(["eu-west-1", "eu-west-2", "eu-central-1"], [])
toset([
  "eu-central-1",
  "eu-west-1",
  "eu-west-2",
])

Here we can see we get our first set back unchanged, this is because there is nothing in our second set. Now if we wanted to remove eu-central-1 from our configuration, we can do it in one of two ways shown below:

> setsubtract(["eu-west-1", "eu-west-2", "eu-central-1"], ["eu-central-1"])
toset([
  "eu-west-1",
  "eu-west-2",
])

> setsubtract(["eu-west-1", "eu-west-2"], ["eu-central-1"])
toset([
  "eu-west-1",
  "eu-west-2",
])

Remember, setsubtract() returns a new set with items only in the first set that are not in the second set. Within our configuration, the first set represents var.regions and the second var.disabled_regions. It makes sense to remove it from the first set (var.regions) and include it within the disabled_regions variable as it makes it clear to practitioners the context of eu-central-1 within the current configuration.

Not Using setsubtract()

Now if we don’t use setsubtract() here we actually get a warning from OpenTofu itself when running tofu plan:

... plan output

╷
│ Warning: Provider configuration for_each matches module
│ 
│   on main.tofu line 4, in module "network":
│    4:   for_each = var.regions
│ 
│ This provider configuration uses the same for_each expression as a module, which means that subsequent removal of elements from this
│ collection would cause a planning error.
│ 
│ OpenTofu relies on a provider instance to destroy resource instances that are associated with it, and so the provider instance must outlive
│ all of its resource instances by at least one plan/apply round. For removal of instances to succeed in future you must structure the
│ configuration so that the provider block's for_each expression can produce a superset of the instances of the resources associated with the
│ provider configuration. Refer to the OpenTofu documentation for specific suggestions.
│ 
│ To destroy this object before removing the provider configuration, consider first performing a targeted destroy:
│     tofu apply -destroy -target=module.network

As the warning says, and what was discussed earlier, a provider must exist to manage the resources it creates. You might guess what will happen next, but we’re going to continue for the sake of exploration… Now if we run a tofu plan once again:

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # module.network["eu-central-1"].aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-central-1a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-central-1"].aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-central-1b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-central-1"].aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
    }

  # module.network["eu-west-1"].aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-1a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-west-1"].aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-1b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-west-1"].aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
    }

  # module.network["eu-west-2"].aws_subnet.subnet["private-1"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-2a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.0.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-1"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-west-2"].aws_subnet.subnet["private-2"] will be created
  + resource "aws_subnet" "subnet" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "eu-west-2b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + tags_all                                       = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "private-2"
        }
      + vpc_id                                         = (known after apply)
    }

  # module.network["eu-west-2"].aws_vpc.vpc will be created
  + resource "aws_vpc" "vpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = (known after apply)
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
      + tags_all                             = {
          + "DeployedWith" = "OpenTofu"
          + "Name"         = "example-vpc"
        }
    }

Plan: 9 to add, 0 to change, 0 to destroy.
╷
│ Warning: Provider configuration for_each matches module
│ 
│   on main.tofu line 4, in module "network":
│    4:   for_each = var.regions
│ 
│ This provider configuration uses the same for_each expression as a module, which means that subsequent removal of elements from this
│ collection would cause a planning error.
│ 
│ OpenTofu relies on a provider instance to destroy resource instances that are associated with it, and so the provider instance must outlive
│ all of its resource instances by at least one plan/apply round. For removal of instances to succeed in future you must structure the
│ configuration so that the provider block's for_each expression can produce a superset of the instances of the resources associated with the
│ provider configuration. Refer to the OpenTofu documentation for specific suggestions.
│ 
│ To destroy this object before removing the provider configuration, consider first performing a targeted destroy:
│     tofu apply -destroy -target=module.network
╵

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: t.plan

To perform exactly these actions, run the following command to apply:
    tofu apply "t.plan"

Now let’s apply the configuration with tofu apply t.plan:

module.network["eu-west-1"].aws_vpc.vpc: Creating...
module.network["eu-west-2"].aws_vpc.vpc: Creating...
module.network["eu-central-1"].aws_vpc.vpc: Creating...
module.network["eu-west-2"].aws_vpc.vpc: Creation complete after 1s [id=vpc-0640c640c]
module.network["eu-west-2"].aws_subnet.subnet["private-1"]: Creating...
module.network["eu-west-2"].aws_subnet.subnet["private-2"]: Creating...
module.network["eu-central-1"].aws_vpc.vpc: Creation complete after 1s [id=vpc-0d49f24a5]
module.network["eu-central-1"].aws_subnet.subnet["private-2"]: Creating...
module.network["eu-central-1"].aws_subnet.subnet["private-1"]: Creating...
module.network["eu-west-2"].aws_subnet.subnet["private-1"]: Creation complete after 0s [id=subnet-03add42e4]
module.network["eu-west-2"].aws_subnet.subnet["private-2"]: Creation complete after 0s [id=subnet-022c938bc]
module.network["eu-west-1"].aws_vpc.vpc: Creation complete after 1s [id=vpc-0898dae2b]
module.network["eu-west-1"].aws_subnet.subnet["private-1"]: Creating...
module.network["eu-west-1"].aws_subnet.subnet["private-2"]: Creating...
module.network["eu-central-1"].aws_subnet.subnet["private-1"]: Creation complete after 1s [id=subnet-079cab1ae]
module.network["eu-central-1"].aws_subnet.subnet["private-2"]: Creation complete after 1s [id=subnet-095d1eefd]
module.network["eu-west-1"].aws_subnet.subnet["private-1"]: Creation complete after 0s [id=subnet-08815802c]
module.network["eu-west-1"].aws_subnet.subnet["private-2"]: Creation complete after 0s [id=subnet-08fe30356]
╷
│ Warning: Provider configuration for_each matches module
│ 
│   on main.tofu line 4, in module "network":
│    4:   for_each = var.regions
│ 
│ This provider configuration uses the same for_each expression as a module, which means that subsequent removal of elements from this
│ collection would cause a planning error.
│ 
│ OpenTofu relies on a provider instance to destroy resource instances that are associated with it, and so the provider instance must outlive
│ all of its resource instances by at least one plan/apply round. For removal of instances to succeed in future you must structure the
│ configuration so that the provider block's for_each expression can produce a superset of the instances of the resources associated with the
│ provider configuration. Refer to the OpenTofu documentation for specific suggestions.
│ 
│ To destroy this object before removing the provider configuration, consider first performing a targeted destroy:
│     tofu apply -destroy -target=module.network
╵

Apply complete! Resources: 9 added, 0 changed, 0 destroyed.

Now let’s remove our eu-central-1 element from our var.regions set and rerun tofu plan:

module.network["eu-west-1"].aws_vpc.vpc: Refreshing state... [id=vpc-0898dae2b]
module.network["eu-west-2"].aws_vpc.vpc: Refreshing state... [id=vpc-0640c640c]

Planning failed. OpenTofu encountered an error while generating this plan.

╷
│ Warning: Provider configuration for_each matches module
│ 
│   on main.tofu line 4, in module "network":
│    4:   for_each = var.regions
│ 
│ This provider configuration uses the same for_each expression as a module, which means that subsequent removal of elements from this
│ collection would cause a planning error.
│ 
│ OpenTofu relies on a provider instance to destroy resource instances that are associated with it, and so the provider instance must outlive
│ all of its resource instances by at least one plan/apply round. For removal of instances to succeed in future you must structure the
│ configuration so that the provider block's for_each expression can produce a superset of the instances of the resources associated with the
│ provider configuration. Refer to the OpenTofu documentation for specific suggestions.
│ 
│ To destroy this object before removing the provider configuration, consider first performing a targeted destroy:
│     tofu apply -destroy -target=module.network
╵
╷
│ Error: Provider instance not present
│ 
│ To work with module.network["eu-central-1"].aws_vpc.vpc its original provider instance at
│ provider["registry.opentofu.org/hashicorp/aws"].by_region["eu-central-1"] is required, but it has been removed. This occurs when an element
│ is removed from the provider configuration's for_each collection while objects created by that the associated provider instance still exist
│ in the state. Re-add the for_each element to destroy module.network["eu-central-1"].aws_vpc.vpc, after which you can remove the provider
│ configuration again.
│ 
│ This is commonly caused by using the same for_each collection both for a resource (or its containing module) and its associated provider
│ configuration. To successfully remove an instance of a resource it must be possible to remove the corresponding element from the resource's
│ for_each collection while retaining the corresponding element in the provider's for_each collection.

As expected - it failed! This is why it’s super important to maintain the provider until the resources associated with it has been deleted, after which you can then remove the provider configuration. This is where using setsubtract() is very useful as it:

1. Ensures the provider remains and therefore clears the warning from showing

2. It’s clear to practitioners, at least in our configuration with our variable names, which regions have a deployment and which do not.

Limitations

While this is super awesome, it’s important to highlight the limitations of using for_each at the provider level:

1. You can only use for_each on variables and locals that can be obtained statically. Expressions that rely on data sources or resources are currently not usable.

2. If you have an already-deployed infrastructure, don't simply remove a provider from the list as this will make it impossible for OpenTofu to destroy the infrastructure in this region. Instead, you will need to implement removing that infrastructure first and then remove the provider from the list. See the disabled_regions variable for an example above.

3. Currently, each provider used in a for_each must have an alias. Providers without aliases are not supported for now due to internal technical reasons.

4. There is currently no way to pass a set of providers to a module, you can only pass individual providers.

— https://opentofu.org/blog/opentofu-1-9-0-beta1/

We validated some of these as we tried our implementation:

1. Our region names were statically defined and not defined, through say the data “aws_regions” “region” {} data source.

2. We used setsubtract() to ensure resources associated with a provider can be managed without removing the provider first.

3. We defined our alias of by_region for the aws provider.

It would be cool to learn more around the “internal technical reasons” regarding providers without aliases. But I can remember aliases have been around for a very long time, so I imagine it’s core to the language and would require a major refactor.

Conclusion

That concludes our deep-dive! We compared how our configuration would work for Terraform, and now how it could work with OpenTofu! We’ve configured for_each at the provider level and looked at ways to manage the lifecycle of the resources each provider manages and what you need to bear in mind when using it in your configurations! OpenTofu v1.9.x also adds an -exclude flag, which does the inverse of -include - apply everything except.

You can read more at: https://opentofu.org/blog/opentofu-1-9-0-beta1/ where it expands on the above and formed the reference for this article!