Harnessing Docker Agents: Enhancing Your CI/CD Pipeline Efficiency

Jasai HansdaJasai Hansda
8 min read

Table of contents

In today's fast-paced software development landscape, optimizing the efficiency of your CI/CD pipeline is crucial. Harnessing Docker agents within Jenkins offers a powerful solution by providing a consistent, isolated, and portable environment for your build processes. This approach not only streamlines development workflows but also ensures that applications run smoothly across various stages, from local development to cloud deployment. In this article, we'll explore how Docker agents can enhance your CI/CD pipeline, particularly for deploying a Python web application on AWS EC2.

What is a Docker Agent?

A Docker agent is an environment within a CI/CD pipeline that uses Docker containers to run tasks. It ensures that every stage of the pipeline (build, test, or deployment) executes in an isolated, consistent environment, regardless of the underlying host system.

With Docker agents, developers can:

  • Use pre-defined Docker images as build environments.

  • Avoid dependency mismatches across environments.

  • Maintain consistency across builds, ensuring the pipeline behaves the same on any system.

Why Use Docker Agents in CI/CD?

  1. Consistency: Docker containers ensure that the build and test environments are identical, eliminating the "works on my machine" problem.

  2. Efficiency: Containers are lightweight and can be spun up or destroyed quickly, making CI/CD pipelines faster.

  3. Flexibility: You can use custom Docker images tailored to your application’s needs.

  4. Isolation: Each pipeline stage runs in a clean environment, reducing interference between steps.

Using a Docker Agent in Jenkins for Testing Environment Setup

Test JenkinsFile- basic-flask-app/Test_Jenkinsfile at main · Jasai007/basic-flask-app

The link directs to a Test Jenkins file in GitHub repository. This Jenkinsfile utilizes a Docker agent to test the environment details.

You can use this file as your initial pipeline to directly execute and test the environment setup.

Configure Jenkins to use Docker Agent:

  • Create a new Test Pipeline.

  • Choose Pipeline script from SCM from the Definition dropdown to get the script from GIthub.

  • Provide the script path with the filename. (e.g., if the file name is Jenkinsfile and it is located inside the Test_Jenkinsfile folder, then provide the path as Test_Jenkinsfile/Jenkinsfile.)

  • Navigate to Manage Jenkins > Plugins > Available to search and install Docker Pipeline Plugin.

  • Now we can try building the pipeline by clicking on "Build Now."

Pipeline Structure

pipeline {
  agent {
    docker { image 'node:16-alpine' }
  }
  stages {
    stage('Test') {
      steps {
        sh 'node --version'
      }
    }
  }
}

1. Pipeline Block

  • The pipeline block is the root of the declarative pipeline. It defines the entire workflow.

2. Agent Block

agent {
  docker { image 'node:16-alpine' }
}

  • Specifies the environment where the pipeline will run.

  • Docker agent:

    • Pulls and runs a lightweight Docker image node:16-alpine as the environment for the pipeline steps.

    • This ensures the steps are executed inside a container running Node.js version 16 on an Alpine Linux base.

3. Stages Block

  • Defines the logical steps of the pipeline workflow.

4. Stage: Test

stage('Test') {
  steps {
    sh 'node --version'
  }
}
  • Contains a single stage named Test.

5. Steps block:

  • Runs the command node --version in the Docker container.

  • This step verifies that Node.js is correctly installed in the container and outputs its version.

Execution Flow

  1. Jenkins starts the pipeline.

  2. The Docker image node:16-alpine is pulled (if not already available) and spun up as a container.

  3. Inside the container:

    • The node --version command is executed to check the Node.js version.

  4. The container is stopped and cleaned up once the pipeline finishes.

This specific pipeline is a minimal example, serving as a demonstration or test to ensure Node.js works in the specified Docker image.

Output

The Jenkins pipeline successfully executed using a Docker agent. It started by checking out the Jenkinsfile from the GitHub repository https://github.com/Jasai007/basic-flask-app.git. The pipeline utilized a Docker container with the node:16-alpine image to ensure a consistent environment. During the "Test" stage, it verified the Node.js installation by running the node --version command, which confirmed Node.js version 16.20.2. The pipeline completed without errors, and the Docker container was stopped and removed, indicating a successful build process.

Started by user Jasai Hansda
Obtained Jenkinsfile from git https://github.com/Jasai007/basic-flask-app.git
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/myPipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --resolve-git-dir /var/jenkins_home/workspace/myPipeline/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/Jasai007/basic-flask-app.git # timeout=10
Fetching upstream changes from https://github.com/Jasai007/basic-flask-app.git
 > git --version # timeout=10
 > git --version # 'git version 2.39.5'
 > git fetch --tags --force --progress -- https://github.com/Jasai007/basic-flask-app.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse refs/remotes/origin/main^{commit} # timeout=10
Checking out Revision 0c8a12929a33e458304ae135d88399cd1f6bd222 (refs/remotes/origin/main)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0c8a12929a33e458304ae135d88399cd1f6bd222 # timeout=10
Commit message: "Update Jenkinsfile"
 > git rev-list --no-walk 0c8a12929a33e458304ae135d88399cd1f6bd222 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . node:16-alpine
.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
Jenkins seems to be running inside container f1c03a6c734e45b0d77aee800764ae3bb094bf9169640ec10c76fb3b3c19bbe1
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/myPipeline --volumes-from f1c03a6c734e45b0d77aee800764ae3bb094bf9169640ec10c76fb3b3c19bbe1 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:16-alpine cat
$ docker top 72f25fa0c0458592d74ff04c6c72eee2f5a7183870eaf7862def4ee54d99a5fb -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] sh
+ node --version
v16.20.2
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1 72f25fa0c0458592d74ff04c6c72eee2f5a7183870eaf7862def4ee54d99a5fb
$ docker rm -f --volumes 72f25fa0c0458592d74ff04c6c72eee2f5a7183870eaf7862def4ee54d99a5fb
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Container Lifecycle in Docker Agents

When using Docker agents, containers go through the following lifecycle:

  1. Start: The CI/CD tool spins up a container for the specific stage.

  2. Execute: The pipeline runs tasks (e.g., building, testing) inside the container.

  3. Stop and Remove: Once the tasks are complete, the container is stopped and removed to free up resources.

Benefits of Docker Agents in CI/CD

  • Portability: Build once, run anywhere.

  • Efficiency: Lightweight containers reduce pipeline execution time.

  • Scalability: Easily scale pipelines with parallel container executions.

  • Reproducibility: Ensures consistent environments across builds.

Conclusion

Using Docker agents in CI/CD pipelines offers a reliable and efficient method to build, test, and deploy applications. Containerizing each stage minimizes dependencies and errors, ensuring consistent code behavior across environments.

Implementing Docker agents is essential for modern DevOps workflows. Start using them in your CI/CD process today to see the benefits!

History of Command Usage

history
    1  sudo yum install -y  openjdk-17-jre
    2  wget https://download.oracle.com/java/17/archive/jdk-17.0.11_linux-x64_bin.rpm
    3  sudo yum install -y jdk-17.0.11_linux-x64_bin.rpm
    4  uname -m
    5    sudo yum install -y docker
    6  docker ps
    7   sudo service docker start
    8   sudo usermod -a -G docker ec2-user
    9  docker ps
   10  java -version
   11  docker run -d -p 8080:8080 -p 50000:50000 --restart=on-failure -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts-jdk17
   12  docker ps
   13  cat /var/jenkins_home/secrets/initialAdminPassword
   14  docker exec nifty_vaughan cat /var/jenkins_home/secrets/initialAdminPassword
   15  docker stop 188ebafd79d0
   16  docker run -d -p 8080:8080 -p 50000:50000 --restart=on-failure -v jenkins_home:/var/jenkins_home jenkins jenkins/jenkins:lts-jdk17
   17  docker run -d -p 8080:8080 -p 50000:50000 --restart=on-failure --name jenkins -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts-jdk17
   18  docker ps
   19  docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
   20  docker ps
   21  docker stop 16152a181014
   22  docker ps pa
   23  docker ps -a
   24  docker start 16152a181014
   25  docker ps -a
   26  docker stop 16152a181014
   27   docker run -d     --name jenkins     -p 8080:8080 -p 50000:50000     -v /var/jenkins_home:/var/jenkins_home     -v /var/run/docker.sock:/var/run/docker.sock     jenkins/jenkins:lts
   28   docker run -d     --name jenkins2     -p 8080:8080 -p 50000:50000     -v /var/jenkins_home:/var/jenkins_home     -v /var/run/docker.sock:/var/run/docker.sock     jenkins/jenkins:lts
   29  docker ps
   30   docker exec -it jenkins2 /bin/bash
   31  docker ps -a
   32   docker run -d     --name jenkins3     -p 8080:8080 -p 50000:50000     -v /var/jenkins_home:/var/jenkins_home     -v /var/run/docker.sock:/var/run/docker.sock     jenkins/jenkins:lts
   33  docker ps -a
   34  docker logs 3e575a24b4f8
   35    sudo chown -R 1000:1000 /var/jenkins_home
   36    ls -ld /var/jenkins_home
   37  docker ps -a
   38  docker rm 3e575a24b4f8
   39  docker rm e80b348dd847
   40  docker ps -a
   41   docker run -d     --name jenkins3     -p 8080:8080 -p 50000:50000     -v /var/jenkins_home:/var/jenkins_home     -v /var/run/docker.sock:/var/run/docker.sock     jenkins/jenkins:lts
   42  docker ps -a
   43   docker exec -it jenkins3 /bin/bash
   44  docker stop jenkins3
   45  docker rm jenkins3
   46  docker ps -a
   47  docker start jenkins3
   48  docker start jenkins
   49  curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee   /usr/share/keyrings/jenkins-keyring.asc > /dev/null
   50  echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]   https://pkg.jenkins.io/debian binary/ | sudo tee   /etc/apt/sources.list.d/jenkins.list > /dev/null
   51  sudo apt-get update
   52  sudo apt-get install jenkins
   53  curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee   /usr/share/keyrings/jenkins-keyring.asc > /dev/null
   54  sudo yum install jenkins
   55  docker --version
   56  docker --version
   57  sudo usermod -aG docker jenkins
   58  sudo systemctl restart docker
   59  sudo systemctl restart jenkins
   60  docker --version
   61  docker ps
   62   docker exec -it jenkins /bin/bash
   63  whoami
   64   docker exec -it jenkins /bin/bash
   65  docker exec -u root -it jenkins /bin/bash
   66  docker ps
   67  docker stop jenkins
   68  docker rm jenkins
   69   docker run -d     --name jenkins     -p 8080:8080 -p 50000:50000     -v /var/jenkins_home:/var/jenkins_home     -v /var/run/docker.sock:/var/run/docker.sock     jenkins/jenkins:lts
   70   docker exec -it jenkins /bin/bash
   71  docker exec -it --user root jenkins /bin/bash
   72  docker restart jenkins
   73  docker ps
   74  docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
   75   sudo chmod 666 /var/run/docker.sock
   76  node --version
   77   docker exec -it jenkins /bin/bash
   78  node --version
   79  docker ps
   80  history
0
Subscribe to my newsletter

Read articles from Jasai Hansda directly inside your inbox. Subscribe to the newsletter, and don't miss out.

CI/CD PipelineJenkinsDevopsDevOps JourneyUsing Docker as Jenkins agents

Written by

Jasai Hansda
Jasai Hansda

Software Engineer (2 years) | In-transition to DevOps. Passionate about building and deploying software efficiently. Eager to leverage my development background in the DevOps and cloud computing world. Open to new opportunities!