Azure AKS Hands-On Labs - Azure Container Registry (ACR) Troubleshooting Common Issues

In this hands-on lab, we will guide troubleshooting a real scenario in Azure Container Registry (ACR) for some common issues.

Learning objectives:

In this module, you'll learn how to:

• Identify the issues

• Resolve the issues

📝Prerequisites:

• Azure CLI installed or access to Azure Cloud Shell.

• Docker installed on your local machine.

📝Create an Azure Container Registry (ACR)

• Create a Resource Group

  • In this lab, we’re using variables to set resource group name, acr name and location.

      rg=<resourcegroupname>
      acr=<acrname>
      location=<region>
    

      az group create -n $rg -l $location
    

    

    

• Create an Azure Container Registry (ACR)

    az acr create -g $rg -n $acr --sku Basic
  

  

  

• Login to ACR

    az acr login -n $acr
  

  

• Verify the ACR

    az acr show -n $acr -g $rg
  

  Ensure that the status shows as ProvisioningState: Succeeded.

  

  📝Scenario 1: Unable to Push Images to ACR

  Symptom: When trying to push an image, you receive an "access denied" error.

  Expected Error Message:

    denied: requested access to the resource is denied
  

  Troubleshooting Steps:

  1. Check Login Credentials:

     • Ensure you are logged into the Azure CLI:

         az login
       

       Log into the ACR:

         az acr login -n $acr
       

  2. Verify Role Assignments:

     • Check if the user/service principal has the appropriate role assigned:

         az role assignment list --assignee <UserPrincipalName> --scope /subscriptions/<SubscriptionId>/resourceGroups/$rg/providers/Microsoft.ContainerRegistry/registries/$acr
       

  3. Check Image Tagging:

     • Ensure the image is tagged correctly before pushing:

         docker tag <ImageName>:<Tag> $acr.azurecr.io/<ImageName>:<Tag>
       

  4. Try Pushing Again:

      docker push <RegistryName>.azurecr.io/<ImageName>:<Tag>
     

     

📝Scenario 2: Unable to Pull Images from ACR

Symptom: When attempting to pull an image, you receive a "manifest not found" error.

Expected Error Message:

manifest for <RegistryName>.azurecr.io/<ImageName>:<Tag> not found

Troubleshooting Steps:

1. Check Image Availability:

   • List images in the ACR:

       az acr repository list -n $acr -o table
     

2. Verify Image Tags:

   • List tags for the specific image:

       az acr repository show-tags -n $acr --repository <ImageName> -o table
     

3. Check the Pull Command:

   • Ensure the pull command uses the correct image name and tag:

       docker pull $acr.azurecr.io/<ImageName>:<Tag>
     

     

     

📝Scenario 3: Network Issues Connecting to ACR

Symptom: Timeout or connection errors when trying to push or pull images.

Expected Error Message:

    Error response from daemon: Get https://<RegistryName>.azurecr.io/v2/: dial tcp <IP>:443: i/o timeout

Troubleshooting Steps:

1. Check Firewall and Network Security Group Rules:

   • Ensure that your local machine can access the ACR endpoint. Check your network security group rules in Azure.

2. Test Network Connectivity:

   • Use tools like ping or telnet to check connectivity to the ACR:

       ping <RegistryName>.azurecr.io
       telnet <RegistryName>.azurecr.io 443
     

     

3. Inspect Azure Service Health:

   • Check for any ongoing outages or service disruptions in the Azure portal.

📝Scenario 4: Image Not Found After Push

Symptom: After pushing an image, it is not found when listing the images.

Expected Behavior: The image should appear in the list after a successful push.

Troubleshooting Steps:

1. Verify Push Success:

   • Check the output of the push command for errors.

2. List Repositories and Tags:

   • Run the following commands to check if the image is there:

       az acr repository list -n $acr -o table
       az acr repository show-tags -n $acr --repository <ImageName> -o table
     

3. Check for Unauthenticated Pushes:

   • If you pushed without authentication, check the az acr login command or check the credentials used for the push.

📝Scenario 5: Insufficient Quota or Resource Limits

Symptom: Unable to push more images due to quota limits.

Expected Error Message:

    denied: quota exceeded

Troubleshooting Steps:

1. Check ACR Limits:

   • Review the limits and quotas for your ACR tier:

       az acr show -n $acr --query "sku" -o table
     

2. Delete Unused Images:

   • If you are near your limit, delete unused images:

       az acr repository delete -n $acr --repository <ImageName> --tag <Tag> --yes
     

3. Upgrade ACR Tier:

   • If necessary, consider upgrading your ACR tier to a higher SKU:

       az acr update -n $acr --sku Premium
     

📌Note - At the end of each hands-on Lab, always clean up all resources previously created to avoid being charged.

Congratulations — This lab guide covers several common issues encountered when using Azure Container Registry and provides troubleshooting steps to resolve them. By following these steps, you can effectively troubleshoot and resolve problems with ACR, ensuring a smoother workflow for your container images.

Thank you for reading. I hope you understood and learned something helpful from my blog.

Please follow me on Cloud&DevOpsLearn and LinkedIn, franciscojblsouza