Streamlining Your Build Process with Docker on Azure Pipelines

Introduction

Modern software teams frequently grapple with inconsistent environments, dependency conflicts, and slow build pipelines. These challenges can lead to failed builds, deployment bugs, and significant troubleshooting overhead. By adopting Docker containerization alongside Azure Pipelines, you can create a reliable, repeatable, and efficient CI/CD workflow.

The Challenge

• Environment Drift: "Works on my machine" syndrome due to differing dependencies and OS configurations.

• Dependency Hell: Conflicting library versions causing build failures.

• Slow Builds: Inefficient resource utilization and lack of caching leading to longer build times.

• Security Risks: Running builds in root contexts exposing vulnerabilities.

The Solution

Docker containers encapsulate applications and their dependencies into isolated, reproducible units. When combined with Azure Pipelines’ cloud-hosted agents, you gain:

• Consistent Build Environment: Same OS, libraries, and tools across all stages.

• Layered Caching: Faster incremental builds by reusing unchanged layers.

• Secure Isolation: Minimal base images and non-root containers reduce attack surface.

• Scalability & Portability: Deploy containers anywhere—on Azure Kubernetes Service (AKS), virtual machines, or local datacenters.

Overview of Azure Pipelines & Docker Integration

Azure Pipelines is a fully managed CI/CD service offering parallel job execution, YAML pipeline as code, and seamless integration with Git repos. Integrating Docker enables:

1. Automated Container Builds: Build and push Docker images to Azure Container Registry (ACR) or Docker Hub.

2. Multi-Stage Pipelines: Define distinct stages—build, test, security scan, and deploy—in one YAML file.

3. Infrastructure as Code: Use Bicep or Terraform tasks within your pipeline to provision resources.

Pro Tip: Use Microsoft-hosted Ubuntu agents for quick starts, then move to self-hosted RHEL agents for full control.

Setting Up Docker on a Self-Hosted RHEL Agent

Prerequisites

• Self-hosted agent: RHEL 8/9 with sudo privileges.

• Azure DevOps PAT: Scope includes Agent Pools and ACR permissions.

• Network Access: Outbound Internet or corporate proxy.

Step 1: Install Docker Engine & Compose

# Add Docker CE repository
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

# Install Docker packages
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Enable and start Docker
sudo systemctl enable docker --now
systemctl start docker

Step 2: Configure Docker Daemon & Permissions

1. Custom Data Directory:

    echo '{"data-root":"/app/docker"}' | \
      sudo tee /etc/docker/daemon.json
    sudo systemctl restart docker
   

2. Non-Root Usage:

    sudo usermod -aG docker udeploy
    newgrp docker
   

Creating & Registering Your Self-Hosted Agent

Follow Microsoft’s self-hosted agent docs for full details. You can also follow my detailed guide on how to set up build agents on RHEL here: How to Set Up a Build Agent on RHEL. follow from Step 8

Testing the Installation

Run the following commands to verify that Docker is installed and functioning:

• Check the Docker version:

    docker --version
  

• Run a test container:

    docker run hello-world
  

The Importance of Docker in Azure Pipelines

Consistent Environments

Docker ensures that applications run in identical environments across development, testing, and production, eliminating the "it works on my machine" problem. This consistency improves collaboration among team members and reduces the likelihood of environment-specific bugs.

Dependency Management

By isolating applications and their dependencies, Docker simplifies the management of software dependencies, reducing conflicts and errors. With Docker images, developers can specify exact versions of dependencies, ensuring that all stages of the pipeline use the same setup.

Faster Build and Deployment

Docker's lightweight containers enable faster build times and efficient resource utilization, speeding up the overall CI/CD process. Reusable layers in Docker images further reduce the time required for incremental builds, making the pipeline more agile.

Security

Docker enhances security by isolating containers and limiting exposure to vulnerabilities. Using minimal base images and automated scanning tools can further improve container security. Additionally, role-based access control (RBAC) and network policies within Docker allow fine-grained control over resources and processes.

Scalability

Docker's compatibility with orchestration tools like Kubernetes enables seamless scaling of applications. In Azure Pipelines, containerized applications can be easily deployed across multiple environments, adapting to varying workloads without additional configuration.

Portability

Docker containers are platform-agnostic, meaning they can run on any system that supports Docker. This portability ensures smooth transitions between different stages of the pipeline and simplifies disaster recovery processes.

Resource Optimization

By sharing the host system's kernel and using fewer resources than virtual machines, Docker allows efficient utilization of system resources. This optimization is particularly beneficial for CI/CD pipelines that need to handle multiple builds and tests simultaneously.

Conclusion

By integrating Docker with Azure Pipelines on a self-hosted RHEL agent, you achieve faster builds, consistent environments, and secure automation. Follow this guide to optimize your CI/CD workflows and stay ahead in DevOps best practices.

Enjoyed this article? Subscribe to our newsletter for more such blogs