Deploying Terraform in a GCP Cloud Build Pipeline

If you followed my last post Getting started with Terraform on GCP, you've hopefully started deploying your GCP infrastructure using Terraform. Great start! If you haven't yet, I recommend checking out my post if you're new to IaC, DevOps, Terraform etc.

In this tutorial, we'll look at running our Terraform in a pipeline, specifically in GCP Cloud Build. Apologies this has been so delayed after my first part, I've been swamped and enjoying some nice weather. This tutorial is a little rough and ready, time was the priority here.

Disclaimer! This demo is intended to quickly get you using a pipeline to deploy Terraform and demonstrate the benefits, this method is not fit for production and should not be used for so!

I will write up a quick summary of how to improve this and where you can go afterwards.

What is a pipeline? And more so, why bother?!

You have probably heard the term pipeline and CI/CD.

A pipeline is usually part of the CI/CD process. CI/CD is the continuous integration of developer code and when that code has been developed and merged it is then deployed into the working environment. continuous building, integration, testing and deployment.

Running our Terraform in a pipeline makes it more transparent and collaborative. No more wondering who is pushing to what and running Terraform from their laptop. It also ensures the tasks running are running in a consistent environment, the same version of providers, Terraform etc.

We'll look at the different parts of what makes our pipeline and how we get it to run successfully.

Getting started

First, create a new project which you don't mind deleting afterwards. If you're unable to, just make a note of everything you add ad create so you can delete it when you're done to avoid any costs or security implications.

We'll need to make sure Cloud Build is enabled. Then we'll give the cloud build service account the editor IAM role for this demo purpose.

In real-world circumstances you'd need to be using least privilege IAM, using a dedicated service account to deploy Terraform and only the IAM roles that it needs (This is best practice, I repeat! In real-world projects use least privilege when assigning IAM roles!).

We'll need also to add the build yaml files that will tell Cloud Build what tasks we want to run. You can find it here.

GCP Build Trigger

Create a GCP Cloud Build Trigger. We'll need to connect Cloud Build to our GitHub repo so it can map and allow builds.

Then we'll configure our trigger to run our Terraform plan when we create a pull request to the main branch, the build config will point to the terrform-plan.yaml file we created earlier.

We'll also create one for running terraform apply, the difference in this trigger will be "Push to a branch" As we want this trigger to invoke when the pull request has been approved and merged into the main branch.

So, let's push to our feature branch. We can make an arbitrary change, like changing the name of the VM. Commit and push to our feature branch and create a Pull Request.

Cool! Our build has been triggered and the terraform-plan trigger is running, it will also appear as a check, if the check fails, if the Terraform plan fails for whatever reason, we won't be able to merge our "broken" code to the main branch. So adds a good safety net to our repo.

Our Terraform plan should pass with resources to add, we can now merge to main which will effectively push to main and trigger our terraform-apply Cloud Build trigger and apply our Terraform.

We can check the progress of the build from the Cloud Build log output from the Cloud Build dashboard, watch for any errors etc.

Build Success!

So that's that! You've just automated your Infrastructure as code! Congrats! It seems like a lot to do to effectively just run terraform apply, but hopefully, you'll start to see the benefits of deploying your infrastructure in a more automated way. So hopefully this has helped even just a bit.

Clean up

Don't forget to delete any service accounts, and service account keys, and remove any editor IAM roles if you're using a project you intend on keeping. Or to be safe just delete your project.

Ideas for improvement

So now you can see where a pipeline and automating your IaC deployments can be beneficial.

But there are a few security implications with this method and some security considerations you should make when taking this to a real-world and production-grade environment.

Use a dedicated service account for terraform, adding the Cloud Build default service account to the editor role is not a good idea, it's too broad an IAM role and everyone with the right role can use it.

You could also use that service account to trigger your builds instead of the default cloud Build service account.

There are some additional logging options that need to be added to the cloudbuild.yaml files. Or, you can use account impersonation as part of your cloudbuild.yaml. could build acting as your Terraform service account. More on account impersonation here.

Remote state. In this demonstration our state was not persistent, we let Cloud Build run terraform init locally, meaning we would have lost the terraform state file when the Cloud Build container environment ended.

We really should have created a GCS bucket and added a backend.tf file to tell terraform to store the state file in our remote GCS bucket. if we wanted to make any changes or destroy, we had no reference to state! When I get time I'll update this demo and the repo to add a backend.tf file.

For you more eagle-eyed readers, yes I did accidentally commit the name of a service account and project ID. They are long gone now and absolutely not a good idea to commit to a public repo!

Some pre-commit hooks could have probably saved me from the embarrassment, there are some really good ones for Terraform projects, Google search is your friend!

Any comments?! Please let me know! I'm always interested in hearing about different approaches to pipelines and deploying infrastructure. There really are so many different ways, approaches and opinions to this, all with different weighted pros and cons.

Please ping me if you have any questions or feedback! Good or bad!

Thanks for reading and following along!