Part 3: Continuous Deployment with GitOps Using Argo CD and AKS

1. Introduction

The Continuous Deployment (CD) phase of this project ensures that any new Docker image built and pushed during the CI stage is automatically deployed to the Kubernetes cluster (AKS). Using a GitOps approach with Argo CD, the system continuously monitors the Git repository for changes in the Kubernetes deployment YAML files. When a new image tag is updated in these manifests—via an automated update stage—Argo CD detects the change and synchronizes the desired state from Git with the actual state in the AKS cluster. This process enables seamless, automated, and version-controlled deployments of all microservices.

2. GitOps Approach

2.1. Git as a Single Source of Truth:

GitOps treats Git as the definitive source for all deployment configurations. In my case, the Git repository is hosted on Microsoft Azure, and GitOps continuously monitors this repository for any changes — a process called continuous reconciliation.

2.2. Automatic Drift Detection and Correction:

If someone changes any file, such as updating an image tag version or modifying secrets/configurations, this creates a drift between the live system and the repository. GitOps detects this drift and automatically corrects it within seconds, ensuring the system state stays consistent. This process is known as drift detection and drift fix.

3. Update Stage in CD Pipeline

3.1. Decoupling CI and CD

Finally, I have the Docker images pushed to our Azure Container Registry (ACR). Now, the second stage begins: ensuring that the new image is deployed to the Azure Kubernetes Service (AKS). But how does this happen?

There is no direct connection between the Continuous Integration (CI) and Continuous Deployment (CD) stages because Argo CD monitors the Git repository—not the ACR. To solve this, I write a script and add it to the Git repository. This script updates the Kubernetes manifest files—specifically the deployment YAML files—with the new image version. For example, it updates the image tag from version v86 to v87.

Once the manifest file is updated, Argo CD detects the change and synchronizes it with the Kubernetes cluster. Kubernetes then knows there is a new image version for, say, the voting microservice, and it redeploys the updated container accordingly. This is essentially how the automated deployment happens.

To add the update stage, I create a new stage in the microservice’s YAML pipeline—similar to how I added the build and push stages before. I name this stage Update. Inside it, I define a job named Update and add a Shell Script task to run the script updateK8sManifests.sh, which updates the deployment files located in the k8s-specification folder.

This script requires three arguments:

1. The microservice name (e.g., voting, since this is the voting microservice pipeline).

2. The container registry name, passed as the variable $(imageRepository).

3. The image tag, which will be updated automatically.

  - stage: Update
    displayName: Update Kubernetes Manifests
    dependsOn: Push
    jobs:
      - job: Update
        displayName: Update Deployment YAML
        steps:
          - task: ShellScript@2
            displayName: Run Update Script
            inputs:
              scriptPath: 'scripts/updateK8sManifests.sh'
              args: 'vote $(imageRepository) $(tag)'

3.2. Explanation of the shell script

Now, I will explain the shell script I use in the update stage.

The script starts with set -x to enable debug mode. This prints each command before it executes, which helps with debugging.

Next, the script takes the repository URL as input. This is necessary because if any changes need to be made in the repository, the script needs to know where to clone it from.

Then, the script clones the GitHub repository onto the agent running the CI/CD pipeline. Of course, the agent must be running for this to work; otherwise, nothing will execute.

After cloning, the repository is copied to a temporary folder. The script uses the cd command to enter this temporary directory (temp_repo) where it executes further commands.

The next important command uses sed to modify the Kubernetes manifest files. Specifically, it updates the image tag in the deployment YAML file. For example, the command needs the container registry name (passed as $2), the microservice name (passed as $1, which helps the script know whether to update the voting, result, or worker deployment files), and the new build tag (passed as $3).

After modifying the deployment files, the script commits the changes and pushes them back to the repository.

#!/bin/bash

set -x

# Set the repository URL
REPO_URL="https://Ad0ra31EzHJOvoJkehmlF0FYc28EseuncIgVxg7LUnpgvjLTijGlJQQJ99BEACAAAAAAAAAAAAASAZDO3RRB@dev.azure.com/jawheramiri/voting-app/_git/voting-app"

# Clone the git repository into the /tmp directory
git clone "$REPO_URL" /tmp/temp_repo

# Navigate into the cloned repository directory
cd /tmp/temp_repo

# Make changes to the Kubernetes manifest file(s)
# For example, let's say you want to change the image tag in a deployment.yaml file
#sed -i "s|image:.*|image: jawhercid/$2:$3|g" k8s-specifications/$1-deployment.yaml
sed -i "s|image:.*|image: jawhercid.azurecr.io/$2:$3|g" k8s-specifications/$1-deployment.yaml

# Add the modified files
git add .

# Commit the changes
git commit -m "Update Kubernetes manifest"

# Push the changes back to the repository
git push

# Cleanup: remove the temporary directory
rm -rf /tmp/temp_repo

4. Provisioning the AKS Cluster

4.1. AKS Cluster creation

To provision the AKS cluster, I began by creating a new cluster named azuredevops in the US West 2 region. I selected an existing resource group called azurecid and ensured that a public IP address would be available and open for all nodes, allowing external communication as needed.

For the node pool configuration, I selected an existing pool but had to change the node type, since the default node type is not supported in the US West 2 region.

I configured the cluster to use auto-scaling, setting the minimum number of nodes to 1 and the maximum to 2. Additionally, I set the maximum number of pods per node to 30, which provides sufficient capacity for the expected workload in this deployment.

4.2. Connecting to AKS

Connecting to AKS involves a few straightforward steps. First, install the Azure CLI by running:

sudo dnf install azure-cli

Next, I connect to the AKS cluster using this command:

az aks get-credentials --name azuredevops --resource-group azurecicd --overwrite-existing

Here, the --name flag specifies the AKS cluster name, and the --resource-group flag specifies the resource group.

To verify the connection, I install the Kubernetes CLI tool kubectl, which is part of the kubernetes-client package:

sudo dnf install kubernetes-client

Finally, I verify the connection by running:

kubectl get pods

5. Installing and Configuring Argo CD

5.1. Argo CD installation steps

To install Argo CD, I began by following two simple commands from the official documentation. First, I created the argocd namespace, and then I applied the Argo CD installation manifest to deploy all the required pods. Once the deployment was complete, I verified that all pods were running successfully.

After installation, I retrieved the argocd-initial-admin-secret, which contains the default admin password. I decoded the base64-encoded value to obtain the actual password—while the default username is simply admin.

To enable external access to the Argo CD web interface, I modified the argocd-server service type from ClusterIP to NodePort. This allowed me to expose Argo CD on a static port. I then retrieved both the external IP address of the Kubernetes node and the assigned NodePort. With this information, I was able to access the Argo CD UI directly in the browser.

Argo

Initially, the ArgoCD UI did not load, so I realized I needed to open the inbound traffic on the specific NodePort in the network security group to allow access to the GUI.

Finally, to connect ArgoCD with the Git repositories hosted in Azure DevOps, I navigated to ArgoCD’s settings and selected "Connect Repositories." I copied the HTTPS URL of the repository into ArgoCD, but to grant proper access, I needed to provide authentication credentials.

To do this, I created a personal access token in Azure DevOps via the Settings > Personal Access Tokens section. Then, I replaced the organization name in the repository URL with the token credentials and validated the connection. This method allows ArgoCD to securely connect and synchronize with the Azure DevOps repositories.

5. 2. Connecting Argo CD with AKS

To connect Argo CD with the AKS cluster, I began by creating a new application within the Argo CD interface. During configuration, I specified the application name and project, and left synchronization settings at their default values. The repository URL was populated automatically, and I set the path to the k8s-specification folder, which contains all the Kubernetes deployment and service YAML files. The cluster URL also populated automatically, and I selected it before applying the configuration.

However, the deployment failed immediately after applying the configuration. Argo CD returned the following error:

InvalidSpecError: Namespace for db /v1, Kind=Service is missing

This same error appeared for all other microservices as well. The issue was that the Kubernetes manifests did not specify a namespace, so Argo CD didn’t know where to deploy the resources.

To resolve this, I added a namespace field under the metadata section of each deployment and service YAML file in the manifest directory. I set the namespace to default in each case and ensured consistency across all resource definitions. Once this change was committed, Argo CD successfully redeployed the application without any further issues.

5.3. Creating Kubernetes ACR Secret

After launching the pipeline, the CI and CD parts should normally work fine, and any change I make should trigger the pipeline. However, after running the pipeline, I encountered an issue—not in the CI part, but in the CD part within Argo CD.

The microservice showed an ImagePullBackOff error. This error means Kubernetes does not have the proper permission to pull the new Docker image from the Azure Container Registry (ACR).

To fix this, I created a secret in the default namespace of the Kubernetes cluster. First, I went to the Azure Container Registry in the Microsoft Azure portal and created access keys. Then, I copied the username and password into the secret configuration file in Kubernetes. The secret is named acr-secret, with the namespace set to default, and the registry server specified. This secret allows Kubernetes to authenticate and pull the Docker images from ACR successfully.

kubectl create secret docker-registry <secret-name> \
    --namespace <namespace> \
    --docker-server=<container-registry-name>.azurecr.io \
    --docker-username=<service-principal-ID> \
    --docker-password=<service-principal-password>

5.4. Fixing CrashLoopBackOff Issues

At this point, Kubernetes should be able to pull the images from the Azure Container Registry (ACR), and everything should work smoothly. However, after launching the pipeline, Argo CD showed that the vote pod was in a CrashLoopBackOff state and marked as unhealthy.

To investigate, I checked the Kubernetes cluster pods and noticed the vote pod’s status was CrashLoopBackOff. This error can have many causes and is often one of the most frustrating errors to debug.

I examined the pod logs using the command:

kubectl logs vote-7fdbddb48b-p6s7c

The logs showed the following error:

exec /usr/local/bin/gunicorn: exec format error

This indicates that the container tried to run the executable gunicorn (a web server used by Python applications), but the kernel could not execute it because the file’s architecture is incompatible with the node’s processor architecture.

Upon checking the AKS cluster node details, I found that the node architecture was arm64, while the container image was built for amd64 architecture. This mismatch caused the CrashLoopBackOff error.

6. Rebuilding Containers for arm64 Architecture

6.1. Using QEMU and Docker Buildx

To resolve this issue, I rebuilt the container image targeting the arm64 architecture, matching the node’s architecture. This was done by adding the following argument in the pipeline YAML:

arguments: '--platform linux/arm64 --no-cache'

stages:
 - stage: Build
   displayName: Build
   jobs:
     - job: Build
       displayName: Build ARM64 Image
       steps:
         - script: |
             docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
             docker buildx create --use
           displayName: Enable QEMU and Buildx


         - task: Docker@2
           displayName: Build ARM64 Image
           inputs:
             containerRegistry: '$(dockerRegistryServiceConnection)'
             repository: '$(imageRepository)'
             command: 'build'
             Dockerfile: 'vote/Dockerfile'
             tags: '$(tag)'
             arguments: '--platform linux/arm64 --no-cache'

After rebuilding the containers for the correct architecture, the application deployed successfully. The Argo CD UI confirms that all microservices are healthy and synchronized, indicating that the infrastructure and deployment pipelines are now working as expected.

Additionally, the full CI/CD pipeline is functioning end to end. From code push to image build, manifest update, and automated deployment, every stage completes without errors—demonstrating a fully operational GitOps workflow using Argo CD and AKS.

6.2. Architecture Comparison: arm64 vs amd64

While deploying my application to AKS, I ran into a exec format error. After some investigation, I discovered that my container was built for the amd64 architecture, but the AKS node was running arm64.

The amd64 (or x86_64) architecture is the standard for most desktops, laptops, and traditional servers. It’s widely supported and is the default for most Docker builds. On the other hand, arm64 (or aarch64) is a 64-bit ARM architecture, originally used in mobile devices but now common in cloud environments due to its efficiency and lower cost.

These two architectures aren’t compatible. If you build a container for amd64 and try to run it on an arm64 node, it won’t work—hence the error.

To fix this, I rebuilt the Docker images specifically for arm64 using Docker Buildx and QEMU, tools that support cross-platform builds. Once the architecture matched the node, the containers ran without issue.

The key takeaway: in Kubernetes, always make sure your image architecture matches your cluster’s node architecture.

7. Summary

In Part 3, the project transitions from Continuous Integration to Continuous Deployment using a GitOps approach with Argo CD and Azure Kubernetes Service (AKS). The update stage is added to the CI/CD pipeline to modify deployment manifests with new Docker image versions. Argo CD continuously monitors these manifests and synchronizes them with the AKS cluster for automated deployment.

The section walks through provisioning AKS, installing and configuring Argo CD, resolving common deployment errors like missing namespaces and image pull failures, and handling CrashLoopBackOff issues due to architecture mismatches. Finally, it explains how to rebuild container images targeting the correct architecture (arm64) using Docker Buildx and QEMU. This ensures compatibility and smooth deployment on the AKS cluster.