AKS with Application Gateway & AGIC (application gate) using Terraform

I'm writing this blog about a recent difficulty I encountered while setting up an AKS cluster. It revolved around being unable to manage the Application Gateway linked to it. A little sneak peek: the issue had to do with identity and permissions.

What is AGIC (Application Gateway IngressController)?

The Application Gateway Ingress Controller (AGIC) is a Kubernetes application, which makes it possible for Azure Kubernetes Service (AKS) customers to leverage Azure's native Application Gateway L7 load-balancer to expose cloud software to the Internet. AGIC monitors the Kubernetes cluster it's hosted on and continuously updates an Application Gateway so that selected services are exposed to the Internet. You can read more about it here.

Prerequisite and scope

In this blog post, we will delve into exploring the utilization of Terraform to accomplish the following tasks:

✅Create a resource group
✅Create a new application gateway
✅Create a new AKS cluster with AGIC add-on enabled
✅Deploy a sample application using AGIC for ingress on the AKS cluster
✅Check that the application is reachable through application gateway

You need to have one vnet and within that atleast two subnets already created in azure . One subnet for AKS cluster and another one for application gateway.

Lets explore the terraform code .

Implement the Terraform code

Create a directory in which to test the sample Terraform code and make it the current directory.

Create a file named providers.tf and insert the following code.

 terraform {

   required_version = "~> 1.2"

   required_providers {
     azurerm = {
       source  = "hashicorp/azurerm"
       version = "~> 3.0"
     }
   }
 }

 provider "azurerm" {
   features {}
 }

Create a file named app-gateway.tf and insert the following code.

 resource "azurerm_resource_group" "rg" {
   name     = var.resource_group_name
   location = var.resource_group_location
 }
 data "azurerm_subnet" "appgwsubnet" {
   name                 = var.application_gateway_subnet_name
   virtual_network_name = var.vnet_name
   resource_group_name  = var.vnet_resource_group_name
 }
 resource "azurerm_public_ip" "test" {
   name                = "publicIp1"
   location            = azurerm_resource_group.rg.location
   resource_group_name = azurerm_resource_group.rg.name
   allocation_method   = "Static"
   sku                 = "Standard"
   tags = var.tags
 }

 resource "azurerm_application_gateway" "network" {
   name                = var.app_gateway_name
   resource_group_name = azurerm_resource_group.rg.name
   location            = azurerm_resource_group.rg.location

   sku {
     name     = var.app_gateway_sku
     tier     = "Standard_v2"
     capacity = 2
   }

   gateway_ip_configuration {
     name      = "appGatewayIpConfig"
     subnet_id = data.azurerm_subnet.appgwsubnet.id
   }

   frontend_port {
     name = local.frontend_port_name
     port = 80
   }

   frontend_port {
     name = "httpsPort"
     port = 443
   }

   frontend_ip_configuration {
     name                 = local.frontend_ip_configuration_name
     public_ip_address_id = azurerm_public_ip.test.id
   }

   backend_address_pool {
     name = local.backend_address_pool_name
   }

   backend_http_settings {
     name                  = local.http_setting_name
     cookie_based_affinity = "Disabled"
     port                  = 80
     protocol              = "Http"
     request_timeout       = 1
   }

   http_listener {
     name                           = local.listener_name
     frontend_ip_configuration_name = local.frontend_ip_configuration_name
     frontend_port_name             = local.frontend_port_name
     protocol                       = "Http"
   }

   request_routing_rule {
     name                       = local.request_routing_rule_name
     rule_type                  = "Basic"
     http_listener_name         = local.listener_name
     backend_address_pool_name  = local.backend_address_pool_name
     backend_http_settings_name = local.http_setting_name
     priority                   = 100
   }

   tags = var.tags

   depends_on = [azurerm_public_ip.test]

   lifecycle {
     ignore_changes = [
       backend_address_pool,
       backend_http_settings,
       request_routing_rule,
       http_listener,
       probe,
       tags,
       frontend_port
     ]
   }
 }

Create a file named aks.tf and insert the following code.

 data "azurerm_subnet" "akssubnet" {
   name                 = var.aks_subnet_name
   virtual_network_name = var.vnet_name
   resource_group_name  = var.vnet_resource_group_name
 }
 resource "azurerm_log_analytics_workspace" "aks_workspace" {
   name                = var.workspace
   location            = azurerm_resource_group.rg.location
   resource_group_name = azurerm_resource_group.rg.name

 }
 resource "azurerm_kubernetes_cluster" "aks" {
   name                                = var.cluster_name
   kubernetes_version                  = var.kubernetes_version
   location                            = var.location
   resource_group_name                 = azurerm_resource_group.rg.name
   dns_prefix                          = var.cluster_name
   private_cluster_enabled             = true
   http_application_routing_enabled    = false
   azure_policy_enabled                = true
   private_cluster_public_fqdn_enabled = var.private_cluster_public_fqdn_enabled
   oms_agent {
     log_analytics_workspace_id = azurerm_log_analytics_workspace.aks_workspace.id
   }

   default_node_pool {
     name                = var.node_pool_name
     node_count          = var.system_node_count
     vm_size             = var.vm_size
     type                = var.aks_node_pool_type
     enable_auto_scaling = var.enable_auto_scaling
     os_disk_size_gb     = var.os_disk_size_gb
     vnet_subnet_id      = data.azurerm_subnet.akssubnet.id
   }

   ingress_application_gateway {
     gateway_id = azurerm_application_gateway.network.id
   }

   azure_active_directory_role_based_access_control {
     managed            = true
     azure_rbac_enabled = true
   }

   identity {
     type = var.identity_type
   }

   network_profile {
     network_plugin    = var.network_plugin
     load_balancer_sku = var.loadbalancer_sku
     network_policy    = var.network_policy
   }
   depends_on = [
     azurerm_application_gateway.network
   ]
 }
 resource "azurerm_log_analytics_solution" "container_insights" {
   solution_name         = var.container_insights
   location              = azurerm_resource_group.rg.location
   resource_group_name   = azurerm_resource_group.rg.name
   workspace_resource_id = azurerm_log_analytics_workspace.aks_workspace.id
   workspace_name        = azurerm_log_analytics_workspace.aks_workspace.name

   plan {
     publisher = var.aks_publisher
     product   = var.aks_product
   }
 }

Create a file named locals.tf and insert the following code.

 # Locals block for hardcoded names
 locals {
   backend_address_pool_name      = "${var.vnet_name}-beap"
   frontend_port_name             = "${var.vnet_name}-feport"
   frontend_ip_configuration_name = "${var.vnet_name}-feip"
   http_setting_name              = "${var.vnet_name}-be-htst"
   listener_name                  = "${var.vnet_name}-httplstn"
   request_routing_rule_name      = "${var.vnet_name}-rqrt"
   app_gateway_subnet_name        = data.azurerm_subnet.appgwsubnet.name
 }

Create a file named roles.tf and insert the following code.

 resource "azurerm_role_assignment" "Network_Contributor_subnet" {
   scope                = data.azurerm_subnet.appgwsubnet.id
   role_definition_name = "Network Contributor"
   principal_id         = azurerm_kubernetes_cluster.aks.ingress_application_gateway[0].ingress_application_gateway_identity[0].object_id
 }

 resource "azurerm_role_assignment" "rg_reader" {
   scope                = azurerm_resource_group.rg.id
   role_definition_name = "Reader"
   principal_id         = azurerm_kubernetes_cluster.aks.ingress_application_gateway[0].ingress_application_gateway_identity[0].object_id
 }

 resource "azurerm_role_assignment" "app-gw-contributor" {
   scope                = azurerm_application_gateway.network.id
   role_definition_name = "Contributor"
   principal_id         = azurerm_kubernetes_cluster.aks.ingress_application_gateway[0].ingress_application_gateway_identity[0].object_id
 }

Create a file named variables.tf and insert the following code.


 variable "resource_group_location" {
   default     = "centralus"
   description = "Location of the resource group."
 }

 variable "app_gateway_name" {
   description = "Name of the Application Gateway"
   default     = "ApplicationGateway1"
 }

 variable "app_gateway_sku" {
   description = "Name of the Application Gateway SKU"
   default     = "Standard_v2"
 }

 variable "app_gateway_tier" {
   description = "Tier of the Application Gateway tier"
   default     = "Standard_v2"
 }

 variable "tags" {
   type = map(string)

   default = {
     source = "terraform"
   }
 }
 variable "resource_group_name" {
   type        = string
   description = "RG name in Azure"
   default     = "rg-test"
 }
 variable "location" {
   type        = string
   description = "Resources location in Azure"
   default     = "centralus"
 }
 variable "cluster_name" {
   type        = string
   description = "AKS name in Azure"
   default     = "test-aks"
 }
 variable "kubernetes_version" {
   type        = string
   description = "Kubernetes version"
   default     = "1.26.3"
 }
 variable "system_node_count" {
   type        = number
   description = "Number of AKS worker nodes"
   default     = 1
 }

 variable "vm_size" {
   type        = string
   description = "size of node pool"
   default     = "Standard_DS2_v2"
 }

 variable "node_pool_name" {
   type        = string
   description = "node pool name"
   default     = "testpool"

 }

 variable "enable_auto_scaling" {
   type        = string
   description = "auto scaling node pool"
   default     = "false"

 }
 variable "aks_node_pool_type" {
   type        = string
   description = "aks_node_pool"
   default     = "VirtualMachineScaleSets"
 }

 variable "os_disk_size_gb" {
   type        = number
   description = "disk size in GB"
   default     = 30

 }

 variable "network_plugin" {
   type        = string
   description = "Network plugin "
   default     = "azure"

 }
 variable "network_policy" {
   type        = string
   description = "azure network policy "
   default     = "azure"

 }
 variable "loadbalancer_sku" {
   type        = string
   description = "specified loadbalancer sku type"
   default     = "standard"

 }
 variable "identity_type" {
   type        = string
   description = "identity type"
   default     = "SystemAssigned"

 }
 variable "workspace" {
   type        = string
   description = "The full name of the Log Analytics workspace with which the solution will be linked."
   default     = "aks-workspace"

 }
 variable "container_insights" {
   type        = string
   description = "name of the log analytics solution "
   default     = "AksContainerInsights"

 }
 variable "aks_publisher" {
   type        = string
   description = "The publisher of the solution.For example Microsoft.Changing this forces a new resource to be created."
   default     = "Microsoft"

 }
 variable "aks_product" {
   type        = string
   description = "The product name of the solution.For example OMSGallery/Containers.Changing this forces a new resource to be created."
   default     = "aksContainerInsights"

 }

 variable "aks_subnet_name" {
   description = "Name of AKS Subnet."
   default = "aks"
   type        = string
 }

 variable "vnet_name" {
   description = "Name of VNet."
   default = "test-vnet"
   type        = string
 }

 variable "vnet_resource_group_name" {
   description = "Name of theVnet resource group."
   default = "vnet-rg"
   type        = string
 }

 variable "private_cluster_public_fqdn_enabled" {
   type        = bool
   description = "Disable a public FQDN on a new AKS cluster"
   default     = false
 }

 variable "application_gateway_subnet_name" {
   type        = string
   description = "name of the subnet where application gateway resides"
   default = "default"
 }

You can either change the default values or create a .tfvars file where you can pass the values of your variables for this iac.

Create a file named outputs.tf and insert the following code.

 output "aks_id" {
   value = azurerm_kubernetes_cluster.aks.id
 }

 output "cluster_name" {
   value = azurerm_kubernetes_cluster.aks.name
 }

 output "networkprofile" {
   value = azurerm_kubernetes_cluster.aks.network_profile
 }

 output "private_cluster_enabled" {
   value = azurerm_kubernetes_cluster.aks.private_cluster_enabled
 }

 output "role_based_access_control_enabled" {
   value = azurerm_kubernetes_cluster.aks.role_based_access_control_enabled
 }
 output "azurerm_log_analytics_workspace" {
   value = azurerm_log_analytics_workspace.aks_workspace.name
 }
 output "azurerm_log_analytics_solution" {
   value = azurerm_log_analytics_solution.container_insights.solution_name
 }

 resource "local_file" "kubeconfig" {
   depends_on = [azurerm_kubernetes_cluster.aks]
   filename   = "kubeconfig"
   content    = azurerm_kubernetes_cluster.aks.kube_config_raw
 }

 output "application_ip_address" {
   value = azurerm_public_ip.test.ip_address
 }

Init , Plan and Apply

Run terraform init to initialize the Terraform deployment.

Run terraform plan to create an execution plan.

Run terraform apply to apply the execution plan to your cloud infrastructure.

Verify the results: Test the Kubernetes cluster

Connect with your Kubernetes cluster. Check the accessibility by kubectl get nodes command or any other command for aks.

You can test the ingress with a fun voting game from Microsoft deployed with this manifest. A big thanks to Thomas Thornton and Microsoft Docs for the great deployment.yaml . Here is the deployment manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-back
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-back
  template:
    metadata:
      labels:
        app: azure-vote-back
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: azure-vote-back
        image: mcr.microsoft.com/oss/bitnami/redis:6.0.8
        env:
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 6379
          name: redis
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-back
spec:
  ports:
  - port: 6379
  selector:
    app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-front
  template:
    metadata:
      labels:
        app: azure-vote-front
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: azure-vote-front
        image: mcr.microsoft.com/azuredocs/azure-vote-front:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: REDIS
          value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-vote-front
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: azure-vote-front
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: azure-vote-front
                port: 
                  number: 80

Run kubectl apply -f deployment.yaml

You can get the application_ip_address from output . Go to your browser and paste this ip address . You will be able to see your application running .

Summary

Setting up the Application Gateway Ingress Controller on an existing Application Gateway can be a challenging task if the necessary permissions haven't been properly configured. Resolving these issues can lead to extended periods of troubleshooting, but my aim is that I've now helped you save valuable time and frustration! 🤓

AKS with Application Gateway & AGIC using Terraform

Table of contents