Making Pulumi feel Like Terraform: A Python Devlopers Guide

If you have been using Terraform for more than five minutes, you would have come across variables and replacing default values for instances using a .tfvars file. Below is a snippet of replacing default values for an EC2 Instance using Terraform:

ec2_instances = [
  {
    name        = "jc-ec2-app001"
    role        = "web"
    size        = "t3.micro"
    application = "app"
    ami         = "ami-008687c5b5546727c"
    family      = "windows"
    volumes = [
      {
        device_name           = "/dev/sda1"
        size                  = 30
        volume_type           = "gp3"
        iops                  = 3000
        throughput            = 125
        encrypted             = true
        delete_on_termination = true
      },
      {
        device_name           = "/dev/xvdf"
        size                  = 20
        volume_type           = "gp3"
        iops                  = 3000
        throughput            = 125
        encrypted             = false
        delete_on_termination = true
      }
    ]
  },
]

The first thing I noticed when using Pulumi with Python is that you don’t have the option of setting up a resource with default values and then passing through new values from the yaml file to replace those default values and essentially getting Pulumi to work in a Terraform way.

This is where I build on the dictionary work I wrote about an article ago. If you want to take a look, check out the following link: Playing Around with Dictionaries in Python

For this example, I will make an AWS backup plan with default values and show how we can replace those with new ones from a yaml file.

I first created my arguments class, where I will pass through the new values I wish to replace. I have already put default values into that argument class under a variable called. backup_configs Of type dictionary. This Class will be the Argument class that will be passed on to the BackupPlanCreator
Class, which will do the heavy lifting of making the AWS Backup Vault and the AWS Backup Plan.

A snippet of the code for the argument class is here:

class BackupPlanCreatorArgs:
    def __init__(
        self,
        name: str,
        vault_name: str,
        base_tags: dict,
    ):
        self.name = name
        self.vault_name = vault_name
        self.base_tags = base_tags

        self.backup_configs = {
            "hourly": {
                "name": "bkp-hourly-ret7-jc",
                "rule_name": "hourly-backup-rule",
                "schedule": "cron(0 * * * ? *)",
                "delete_after_days": 7,
                "start_window": 60,
                "completion_window": 120,
                "recovery_point_tags": {
                    "aws_profile": "jc-profile",
                    "aws_region": "us-east-2",
                    "type": "hourly-backup",
                },
                "enable_windows_vss": True,
            },
            "daily": {
                "name": "bkp-daily-ret14-jc",
                "rule_name": "daily-backup-rule",
                "schedule": "cron(0 5 ? * * *)",
                "delete_after_days": 30,
                "start_window": 60,
                "completion_window": 125,
                "recovery_point_tags": {
                    "aws_profile": "jc-profile",
                    "aws_region": "us-east-2",
                    "type": "daily-backup",
                },
                "enable_windows_vss": True,
            },
            "weekly": {
                "name": "bkp-weekly-ret365-jc",
                "rule_name": "weekly-backup-rule",
                "schedule": "cron(0 6 ? * 2 *)",
                "delete_after_days": 365,
                "start_window": 60,
                "completion_window": 120,
                "recovery_point_tags": {
                    "aws_profile": "jc-profile",
                    "aws_region": "us-east-2",
                    "type": "weekly-backup",
                },
                "enable_windows_vss": True,
            },
            "monthly": {
                "name": "bkp-monthly-ret365-jc",
                "rule_name": "monthly-backup-rule",
                "schedule": "cron(0 6 1 * ? *)",
                "delete_after_days": 365,
                "start_window": 60,
                "completion_window": 120,
                "recovery_point_tags": {
                    "aws_profile": "jc-profile",
                    "aws_region": "us-east-2",
                    "type": "monthly-backup",
                },
                "enable_windows_vss": True,
            },
        }

I wrote the following code into the class, which will replace values in what is essentially a nested dictionary with multiple values. I will go through this code and explain what it does.

        (1) self.final_backup_config = self.backup_configs.copy()

           (2) for key, value in new_backup_config_values.items():
                if (
                   (3) isinstance(value, dict) and key in self.final_backup_config
                ):  # isinstance() checks to see if it has been passed a dictionary.
                    print("Found BackUp Config Dictionary Merging Values")
                   (4) for nested_key, nested_value in value.items():
                        self.final_backup_config[key][nested_key] = nested_value
                else:
               (5)     print("Do not have any new values keeping BackUp Config the same.")

Here, I take a copy of the backup_configs variable values. This goes back to my GoLang days; when passing a variable around, Go will make a copy of that variable so you still have the original values if you want to call them later. We will do more on that later.
The items() method I found to be quite clever in Python. Essentially, you can pass a whole dictionary to items() It will return a view object that displays a list of a dictionary’s key-value tuple pairs. Which is precisely what I needed for this project. To take that one step further and start manipulating that data, you can put the items() method into a for loop. I thought this was quite cool and did not require me to start looking at programming algorithms and tricks to manipulate this data.
The isinstance() the function I added because I am double-checking my code here, again, a nifty function that basically checks before continuing that we are working with a dictionary. Its because of this function that you could essentially put all of this into a module or a function, it allows your code to start becoming quite dynamic and perhaps called elsewhere in your project. I will need to do this further down the line.
for nested_key, nested_value in value.items(): As we are working with a nested dictionary, I need to access the nested value of the dictionary, as this is the actual value that the user will change.
print("Do not have any new values keeping BackUp Config the same."). Now, say that we have passed no changed values to this whole for loop. We then need a way to skip through and keep our values. When I first wrote this if/else statement, I thought it would work with changing defaults and that I would not have to write anything more. I thought I had finished the piece of code. Yet I later found that this code is step one in about five more steps to make it all optional.

Right now, we have something that we can pass from the yaml to manipulate the dictionary. If you were to do this every time and not keep your default values, then this would be all you need to do. Yet that’s not very Terraformy. Terraform allows you to either keep the values or change them. Therefore, to achieve that functionality, we need to do more work. Onwards!!

        (1) if new_backup_config_values is not None:
                for key, value in new_backup_config_values.items():
                    if (
                        isinstance(value, dict) and key in self.final_backup_config
                    ):  # isinstance() checks to see if it has been passed a dictionary.
                        print("Found BackUp Config Dictionary Merging Values")
                        for nested_key, nested_value in value.items():
                            self.final_backup_config[key][nested_key] = nested_value
        (2) else:
                print("Do not have any new values keeping BackUp Config the same.")

if new_backup_config_values is not None: The first part we need to put into all of this is an option to handle type None. This is because in our main.py the Pulumi function pulumi.Config().get("backupConfigs") reads from the Yaml file will sometimes have values, or it will return None for no values. Therefore, we need to handle that here.
else: If the function returns no values, then basically, we are keeping our defaults and handing them over to the BackupPlanCreator Class.

Yet, there is still one issue with our code. Our BackupPlanCreatorArgs Class does have a parameter that expects to receive the changed default values. That parameter is called: new_backup_config_values, If you remember rightly from earlier, this is the foundation for our whole for loop. Yet the idea is sometimes we will pass on new values, and sometimes we will not.

How to solve this one…

(1) backup_configs_yaml = pulumi.Config().get("backupConfigs")
  (2) if backup_configs_yaml == None:
            backup_creator_args = backup.BackupPlanCreatorArgs(
                 name="backupPlanCreator",
                 vault_name=vault_name,
             (3) new_backup_config_values=None,
                 base_tags=base_tags,
            )
(4) else:
      (5)  new_backup_config_values = yaml.safe_load(
            backup_configs_yaml
        )  # This safe loads yaml without Embedding Json and having future issues.
        backup_creator_args = backup.BackupPlanCreatorArgs(
            name="backupPlanCreator",
            vault_name=vault_name,
        (6) new_backup_config_values=new_backup_config_values,
            base_tags=base_tags,
        )

backup_configs_yaml = pulumi.Config().get("backupConfigs"): As described earlier, this function is the key to it all. It will either pass the values that we are changing or will pass the type None
if backup_configs_yaml == None: Once I figured that out, I could start building logic around that function and of passing type None
new_backup_config_values=None, If none is passed from the yaml file, then within the if statement logic, I am simply going to call the args Class and pass None. Taking a copy of backup_configs now allows us to pass the original variable through the for loop logic and to the BackupPlanCreator class with no changes. If we had not taken a copy earlier, our lives would have been made a bit harder, and new code would have had to be written to handle this outcome.
else: Well, this becomes self-explanatory, really, and perhaps I am teaching to suck eggs, but if the function pulumi.Config().get("backupConfigs") does pass values, then the code below will be executed.
new_backup_config_values = yaml.safe_load (backup_configs_yaml): This will safely load the yaml into the class and allow our For loop with the items() method to go through and read the new values. I liked this yaml.safeload() function because it means I don’t have to try and put json formatted code into YAML anyone who has been doing this for some time would know that’s an absolute headache. It also keeps my Yaml file very yaml code-standard and easy to read.
new_backup_config_values=new_backup_config_values This passes our new variable values into the class.

Here is a snippet of Yaml code for reference.

  backupConfigs:
    hourly:
      enable_windows_vss: false

That's how you can use Python to build Terraform-like behaviour into Pulumi and make your code more usable for future engineers.

I hope this has been helpful. Until next time,

Happy Coding,

The Cloud Dude.

Making Pulumi Feel Like Terraform: A Python Developer's Guide

Subscribe to my newsletter

Jason

Jason