The Simplest Way of Slack Notification Settings for GCP Billing Budget

Introduction

Budget management for cloud services is a crucial task for cloud administrators. For enterprises, even small improvements can lead to significant cost savings, while for small businesses, every unnecessary cost directly impacts the bottom line. It's essential to minimize wasteful spending whenever possible.

However, unexpected charges can often occur when using cloud services. For example, I recently noticed that some mysterious scans were running intermittently on GCS in my development environment, resulting in unexpected costs.

To catch such issues early, it's essential to set up budget alerts for immediate detection of anomalies.

GCP allows you to set up alerts based on budget thresholds, not only via the GUI but also through Terraform. I tried to define this using the google_billing_budget resource, but ran into mysterious errors related to notification channels. In this article, I'll share the details and solutions I found.

Confusing Restrictions on Notification Methods

According to the Terraform resource documentation, you can specify notification methods using monitoring_notification_channels in the all_updates_rule block.

So, I tried to send notifications to Slack with the following code:

resource "google_billing_budget" "budget" {
  billing_account = data.google_billing_account.account.id
  display_name    = "Billing Budget for ${var.project_id}"

  budget_filter {
    projects = ["projects/${var.project_number}"]
  }

  amount {
    specified_amount {
      currency_code = "USD"
      units         = var.budget_amount
    }
  }

  threshold_rules {
    threshold_percent = 1.0
  }

  all_updates_rule {
    monitoring_notification_channels = [
      google_monitoring_notification_channel.slack_channel.id,
    ]
  }
}

resource "google_monitoring_notification_channel" "slack_channel" {
  display_name = "Slack Channel"
  type         = "slack"
  labels = {
    "channel_name" = "#foobar"
  }
  sensitive_labels {
    auth_token = var.auth_token
  }
}

However, when applying this, I encountered the following unfriendly error:

googleapi: Error 400: Request contains an invalid argument.

After some frustration, I realized the cause while exploring the GUI.

Although the Terraform documentation doesn't mention it, only email notifications are supported for this feature.

A search only turned up this issue, so it seems this limitation isn't widely known.

If You Still Want to Notify Slack

If you still want to send notifications to Slack, there is a way.

While some guides suggest using Pub/Sub and Cloud Functions to relay notifications to Slack, there's now a simpler method:

Slack allows you to generate a dedicated email address that can forward emails to a channel or DM. You can specify this email address in notification_channels to send budget alerts directly to Slack.

There are some limitations, but this method works even on the free Slack plan.

Here's the final code:

resource "google_billing_budget" "budget" {
  billing_account = data.google_billing_account.account.id
  display_name    = "Billing Budget for ${var.project_id}"

  budget_filter {
    projects = ["projects/${var.project_number}"]
  }

  amount {
    specified_amount {
      currency_code = "USD"
      units         = var.budget_amount
    }
  }

  threshold_rules {
    threshold_percent = 1.0
  }

  all_updates_rule {
    monitoring_notification_channels = [
      google_monitoring_notification_channel.slack_email.id,
    ]
  }
}

resource "google_monitoring_notification_channel" "slack_email" {
  display_name = "Slack Email"
  type         = "email"
  labels = {
    // It's recommended to treat this as a secret  
    email_address = var.slack_email 

  }
  force_delete = false
}

With this setup, notifications will be delivered to Slack. Of course, the best outcome is not to receive any alerts at all—let's hope we stay within budget next month!

Lastly, be sure to check out more information from Tanuki!