CI/CD Pipeline for Private AKS: Integrating Azure DevOps, Terraform, Private ACR, Azure SQL DB, Application Gateway, Azure Key Vault, and Storage

This document explains how to set up a CI/CD pipeline using Azure DevOps, Terraform, and different Azure services to deploy a web application on a private AKS cluster. The deployment is automated and ensures secure communication between services.

Prerequisites

Before diving into this project, here are some skills and tools you should be familiar with:

Need to create a PAT Access Token-

• [x] Clone repository for terraform code
  Note: Replace resource names and variables as per your requirement in terraform code

  • Update terraform.tfvars

• [x] Infra as Code

• [x] Application Repo

1. Azure Subscription: An active Azure subscription is required to create and manage resources.

2. Azure DevOps Account: An Azure DevOps account to create and manage pipelines.

3. Terraform: Installed on the local machine or configured in the Azure DevOps pipeline.

4. Azure CLI: Installed on the local machine or configured in the Azure DevOps pipeline.

5. Service Principal: Created in Azure AD for authentication and authorization in the pipeline.

Technologies/Services Used

1. Azure Kubernetes Service (AKS): For deploying the web application in a private cluster.

2. Azure SQL Database: For storing application data.

3. Azure Container Registry (ACR): For storing Docker images.

4. Azure Application Gateway: For load balancing and routing traffic to the AKS cluster.

5. Azure Virtual Network (VNet): For isolating resources and enabling secure communication.

6. Azure Private Endpoint: For secure access to Azure services within the VNet.

7. Azure Key Vault: For storing secrets and sensitive information.

8. Azure Log Analytics Workspace: For collecting and analyzing logs.

9. Terraform: For infrastructure as code to automate resource creation.

10. Azure DevOps: For creating CI/CD pipelines.

Key Points

1. Architecture Overview:

   • The architecture includes multiple VNets for AKS, ACR, and self-hosted agents.

   • Virtual Network Peering is used to enable communication between VNets.

   • Private Endpoints are used for secure access to Azure services.

2. Build Pipeline:

   • The build pipeline includes tasks to build and push a Docker image to ACR.

   • Terraform configuration files are published as pipeline artifacts for use in the release pipeline.

3. Release Pipeline:

   • The release pipeline is linked to the artifacts published by the build pipeline.

   • It includes stages for deploying resources to AKS using Terraform.

   • Auto-scaling and alert configurations are set up for the web app.

4. Terraform Configuration:

   • Separate files for providers, variables, and main configuration.

   • Variables are stored securely and referenced in the configuration files.

   • The configuration includes creating VNets, AKS cluster, ACR, SQL Database, Application Gateway, and Private Endpoints.

5. Service Connections:

   • Azure DevOps service connections are created for authentication with Azure and Docker Registry.

Setting Up the Infrastructure

First, we'll create the necessary virtual machines using terraform code.

Below is a terraform Code:

Once you clone repo and run the terraform command.

$ ls -l
-rw-r--r-- 1 bsingh 1049089  690 Jan 31 15:01 DevOps_UI.tf
-rw-r--r-- 1 bsingh 1049089 6115 Jan 31 15:01 group_lib.tf
-rw-r--r-- 1 bsingh 1049089 3011 Jan 29 09:47 KeyVault_Create_permission.tf
-rw-r--r-- 1 bsingh 1049089  921 Jan 24 13:03 KeyVault-Getdata.tf
-rw-r--r-- 1 bsingh 1049089  675 Jan 30 11:41 output.tf
-rw-r--r-- 1 bsingh 1049089  813 Jan 24 13:03 provider.tf
-rw-r--r-- 1 bsingh 1049089  326 Jan 24 13:03 ssh_key.tf
-rw-r--r-- 1 bsingh 1049089  924 Jan 30 11:29 Storage.tf
-rw-r--r-- 1 bsingh 1049089 4248 Jan 31 20:36 terraform.tfvars
-rw-r--r-- 1 bsingh 1049089 3727 Jan 30 11:58 variable.tf

You need to run the following terraform command.

Now, run the following command.

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply 
# Optional <terraform apply --auto-approve>

Once you run the terraform command, then we will verify the following things to make sure everything is setup via a terraform.

Inspect the Cloud-Init logs:

Once connected to VM then you can check the status of the user_data script by inspecting the log files

# Primary log file for cloud-init
sudo tail -f /var/log/cloud-init-output.log
                    or 
sudo cat /var/log/cloud-init-output.log | more

• If the user_data script runs successfully, you will see output logs and any errors encountered during execution.

• If there’s an error, this log will provide clues about what failed.

Detailed Steps

1. Setting Up the Architecture:

   • Azure Repos: Store the source code and Terraform configuration files.

   • Azure Storage Account: Store the Terraform state file securely.

   • Build Pipeline: Configure tasks to build and push Docker images.

   • Release Pipeline: Configure tasks to deploy resources using Terraform.

2. Configuring the Build Pipeline:

   • Build and Push Docker Image: Use Docker to build the application image and push it to ACR.

   • Publish Terraform Files: Publish Terraform configuration files as pipeline artifacts.

3. Configuring the Release Pipeline:

   • Initialize Terraform: Initialize the Terraform working directory and download the necessary plugins.

   • Apply Terraform Configuration: Apply the Terraform configuration to create resources in Azure.

   • Deploy Web App: Deploy the Docker image to AKS.

   • Configure Auto-Scaling and Alerts: Set up auto-scaling rules and alert notifications.

4. Terraform Configuration Files:

   • Provider Configuration: Define the Azure provider and authentication details.

   • Variable Definitions: Define variables for resource names, locations, and other configurations.

   • Main Configuration: Define the resources to be created, including VNets, AKS cluster, ACR, SQL Database, Application Gateway, and Private Endpoints.

5. Creating Service Connections:

   • Azure Service Connection: Authenticate with Azure for deploying resources.

   • Docker Registry Service Connection: Authenticate with ACR for pushing Docker images.

Step-by-Step Process:

Configure Service Connection

• First, create Service Connection in Azure Devops.

• Once you create a connection, make sure to note down the connection name, as it will be used in the pipeline.

• On the agent machine, ensure you are logged in with Azure and that the connection is active. If not, log in using the following steps.

    az login --use-device-code
  

• Need to Create a service Connect in the pipeline first.

  • Azure Service Connection

    

    

    

    

    

    

Update Secret variable value details.

• Update secret variable value first

  • update servicePrincipalId

  • update servicePrincipalKey

  • update tenantid

  • Step-01. Go to the key vault and update the value (Azure UI)

• Update the other two values in the same way.

  • Step-02. Update the secret in the Library at Azure DevOps

  • Step-03. Link secrets from an Azure key vault as variables..

Update changes in Repo code as per project details.

• Repo (Infra-as-code)

  • Step-01: script file need to be updated from agent-vm folder

    • update the Organization and PAT token

      

  • Step-02: Update Service Principle Name from private-acr Folder

    • To get the Service Principal's name

      • List Service Principals:

        az ad sp list --query <query_string>

        Example:
        az ad sp list --query "[].{Name:displayName, AppId:appId}" --output table

        # filder with name for your service account
        az ad sp list --query "[?starts_with(displayName, 'Azure')].{Name:displayName, AppId:appId}" --output table

• Update the Service Principal name:

  

Build a YAML pipeline.

• Following is the sequence to create the setup using the pipeline.

  • VNet (Network/Subnet)

  • Agent VM

  • Azure Container Registry

  • SQL Database

  • Application Gateway

  • AKS Cluster

• Step-01: Build YAML pipeline as below:

• 

  

  

  

  Adjust the parameters for vNet Job.

• 

• Adjust the parameters for vm Job.

• 

• Adjust the parameters for acr Job.

• 

• Save and run the pipeline.

• 

• Adjust the parameters for db Job.

• 

• Adjust the parameters for appgateway Job.

• 

• Adjust the ssh key in library for AKS cluster

  • SSH public key need to be updated

    

Run the pipeline.

• 

Install SQL package on Agent VM.

• Open a PuTTY session for the virtual machine and install the required SQL package to check SQL connectivity.

  • install mssql-tools

    • sudo apt-get update

    • sudo apt-get install mssql-tools -y

  • Find the sqlcmd executable:

    • ls /opt/mssql-tools/bin/sqlcmd

  • Add the sqlcmd directory to your PATH:

    • nano ~/.bashrc

    • export PATH="$PATH:/opt/mssql-tools/bin"

  • Apply the changes:

    • source ~/.bashrc

• Verify the installation:

  • sqlcmd -S <server_address> -U -P -d <database_name>

    

• Convert SQL cred to base 64

echo "Server=tcp:yourserver.database.windows.net,1433;Initial Catalog=yourdatabase;Persist Security Info=False;User ID=yourusername;Password=yourpassword;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;" | base64

• Decode base64 cred to Normal.

 echo "U2VydmVyPXRjcDpldGlja2V0ZGJzZXJ2ZXIzMDAxMjAyNS5kYXRhYmFzZS53aW5kb3dzLm5ldCwxNDMzO0luaXRpYWwgQ2F0YWxvZz1ldGlja2V0cy1kYjtQZXJzaXN0IFNlY3VyaXR5IEluZm89RmFsc2U7VXNlciBJRD1henVyZXVzZXI7UGFzc3dvcmQ9cGFzc3dvcmRAMTIzO011bHRpcGxlQWN0aXZlUmVzdWx0U2V0cz1GYWxzZTtFbmNyeXB0PVRydWU7VHJ1c3RTZXJ2ZXJDZXJ0aWZpY2F0ZT1G" | base64 -d

Connect AKS cluster on Agent VM.

• Run the following commands to connect AKS cluster

  • Login to your azure account

    az login

• Set the cluster subscription

    az account set --subscription <ID>

• Download cluster credentials

    az aks get-credentials --resource-group rg-devops --name aksdemo --overwrite-existing

Verify Application Accessibility.

• Try to access application gateway IP in the browser and you will below the error message

  

  

Build the second pipeline (Application).

• Build application pipeline

• 

• run the pipeline.

• It will ask for permission and approve it.

• 

• Update the deployment.yaml file as below (SQL Connection)

• 

• AKS deployment is waiting for approval

• 

Troubleshooting.

• I was getting an error imagepullbackoff and noticed that the SQL connection name was having a problem. To retrieve and update the string from Secret.

• Retrieve and Update the Connection String from the Secret.

  • Since you're using a Kubernetes secret for the database connection, check the secret value:

    kubectl get secret db-connection-secret -o jsonpath="{.data.connection-string}" | base64 --decode

• Pipeline status

• 

• Image status

• 

Update the image name in the manifest file.

Congratulations :-) the application is working and accessible.

• Application is accessible now.

  

• Try to create account in website.

• 

• Following are the cred to login into webpage

• 

Pipeline for Cleanup Infra Setup.

• Following are the sequence to delete the setup using pipeline.

  • AKS Cluster

  • Application Gateway

  • Azure Container Registry

  • SQL Database

  • Agent VM

  • VNet

• Here is the 👉Updated pipeline👈

Environment Cleanup:

• As we are using Terraform, we will use the following command to delete ssh_key, Vault and Storage account.

• Run the terraform command.

    Terraform destroy --auto-approve
  

• I was getting below error message while deleting the setup

• 

• Rerun the destroy command again and it will delete ResourceGroup as well.

• 

Conclusion

This project showcases a strong CI/CD pipeline setup using Azure DevOps, Terraform, and various Azure services. By automating the deployment process, it ensures efficient, scalable, and secure deployment of a web application on a private AKS cluster. Using Terraform for infrastructure as code offers flexibility and easy management, making it a valuable approach for modern cloud-based applications.

Ref Link:

• YouTube Link

• Standard and Basic SKU

• Terraform code for azurerm_public_ip

• Terraform code for azurerm_kubernetes_cluster

• AKS cluster_node_pool

• Kubernetes versions details

• Billing FAQs