A Comprehensive Guide to Using Jenkins for DevOps

Table of contents
- 1. Jenkins Introduction
- 2. Jenkins TLDR — What is Jenkins?
- 3. Jenkins Infrastructure - Master Server and Agents
- 4. Jenkins Agents - Permanent and Cloud Based
- 5. FreeStyle Builds and Pipelines
- 6. Setting up Jenkins using Docker
- 7. Jenkins Web GUI Walkthrough
- 8. Creating a Simple FreeStyle Job
- 9. Exploring The Jenkins Filesystem and Workspace
- 10. Freestyle job - running Python scripts with Jenkins
- 11-13. Setting up Docker Cloud Agents and Templates
- 14. Using labels to restrict Jobs to Agents
- 15. Setting Builds to be automatically triggered on commits
- 16. Setting up Declarative Pipelines using Groovy
- 17. Using a Jenkinsfile for pipelines
- 18. Jenkins BlueOcean
- Conclusion
1. Jenkins Introduction
1.1 What is Jenkins?
Jenkins is an open-source Continuous Integration (CI) and Continuous Delivery (CD) server written in Java.
It is widely used to:
Build and test software automatically.
Integrate changes frequently from multiple developers (CI).
Automate the release and deployment process (CD).
Tagline:
Jenkins helps automate parts of software development related to building, testing, and deploying, facilitating continuous integration and continuous delivery.
1.2 History
2004: Created by Kohsuke Kawaguchi while working at Sun Microsystems. Initially called Hudson.
2011: After Oracle acquired Sun, a community-driven fork was created called Jenkins.
Today: Maintained by the Jenkins project under the Continuous Delivery Foundation (CDF).
1.3 Why Jenkins?
Need | How Jenkins Helps |
Frequent builds | Automatically builds every time code is pushed. |
Early bug detection | Run unit tests early and often. |
Faster releases | Automate packaging and deployments. |
DevOps best practices | Orchestrate infrastructure and code deployments together. |
1.4 Jenkins Architecture
Simple overview:
rustCopyEditDeveloper Pushes Code --> Jenkins Master --> Build/Deploy Jobs --> Test Environments/Production
Internal Components:
Scheduler: Queues and schedules jobs.
Executor: Executes the job logic.
Queue: Organizes pending jobs.
Workspaces: Temporary directories where jobs are executed.
2. Jenkins TLDR — What is Jenkins?
2.1 What is Jenkins, really?
At its core, Jenkins is:
A build server.
A job orchestrator.
An automation engine.
A web-based application.
A central integration hub for all your software delivery pipelines.
Put simply:
Jenkins is a programmable robot that listens to your software development activities (like code commits), and automatically carries out tasks for you — such as building, testing, packaging, analyzing, or even deploying your software — exactly the way you instructed it to.
2.2 Why was Jenkins needed?
Before Jenkins (or similar CI servers), teams worked like this:
Developers wrote code.
Manually compiled/runs tests locally.
Someone packaged builds manually.
Testers manually installed and tested.
Operations manually deployed software.
Result:
Mistakes happened frequently.
Deployments were rare and terrifying ("deployment day" meant "everyone stays late").
Bugs went unnoticed for weeks.
Developers were scared to integrate changes.
This is called "Integration Hell."
Jenkins solved this by automating the entire chain, introducing:
Problem | Jenkins Solution |
Slow integration | Automatic build/test on every commit |
Deployment fear | Push-button or auto deployments |
Manual errors | Automated, repeatable pipelines |
Delayed feedback | Immediate alerts on failures |
Thus, Jenkins makes Continuous Integration (CI) and Continuous Delivery (CD) real and reliable.
2.3 How Jenkins fits into DevOps and Modern Software Delivery
In modern DevOps practices, automation is king.
You automate everything that can be automated.
Jenkins is one of the earliest, strongest tools in this automation movement.
In a DevOps toolchain, Jenkins sits here:
cssCopyEdit[Developer Pushes Code] → [Jenkins Automates Build, Test, Package] → [Deploy to Cloud or Server]
Jenkins roles:
Build automation
Testing automation
Code quality checks
Artifact storage
Deployment orchestration
Notification and monitoring integration
It connects source control systems (like GitHub, GitLab, Bitbucket) with:
Build tools (Maven, Gradle, npm)
Testing tools (JUnit, Selenium)
Deployment tools (Ansible, Docker, Kubernetes)
Thus Jenkins becomes the nervous system connecting code, infrastructure, and deployments.
2.4 Jenkins Key Characteristics
Feature | Explanation |
Open Source | Free, active community, huge plugin ecosystem. |
Extendable | 1800+ plugins for every need (build tools, SCMs, notifications, etc). |
Web-Based | Manage through a web browser anywhere. |
Scriptable | Pipelines are written as code (Groovy DSL), version-controlled. |
Highly Customizable | Every part of a build pipeline can be adjusted, extended, or replaced. |
Cross-Platform | Works on Linux, Windows, macOS, and even inside Docker containers. |
Distributed Builds | Master-agent architecture supports distributed, parallel builds. |
2.5 Visual Understanding — Jenkins In Action
Imagine:
You push a new feature branch to GitHub. Jenkins immediately:
Detects the new commit.
Pulls the updated code.
Builds it (compiling code if necessary).
Runs unit tests.
Checks code style with linters.
Builds a Docker image.
Deploys to a staging server.
Sends a Slack message: "✅ Build Success. Preview available here: ..."
All without you lifting another finger after the push.
2.6 Metaphor — Jenkins as a Factory Manager
🔹 Think of software development like a factory.
🔹 Your source code is the raw material.
🔹 Building, testing, packaging are assembly lines.
🔹 Jenkins is the factory manager, standing at the center, commanding robots to:
Assemble parts (compile code).
Quality-check them (unit test).
Paint and package them (artifact building).
Ship them to customers (deployment).
Without Jenkins (or similar automation), your factory workers (developers) would have to walk to every station manually — slow, error-prone, and exhausting.
2.7 TLDR Summary (in layers)
Type | Description |
In One Line | Jenkins automates building, testing, and deploying your software. |
In One Paragraph | Jenkins is a free open-source server that automates software delivery pipelines, allowing developers to integrate code changes, run tests, build artifacts, and deploy code automatically or on-demand, through a web interface or pipelines as code. |
In Detail | Jenkins is the de facto standard for continuous integration/continuous delivery (CI/CD), acting as a modular, scriptable automation hub connecting development activities to infrastructure management, fully customizable through plugins and pipeline scripting, operable in any environment from local laptops to global cloud infrastructures. |
3. Jenkins Infrastructure - Master Server and Agents
3.1 Master (Controller)
The brain of Jenkins.
Responsibilities:
Manage and monitor agents
Provide the web interface
Schedule builds
Dispatch builds to agents
Record build results
Trigger hooks and notifications
Important:
The Master should not be heavily used for running builds itself in large-scale setups. It should be reserved mostly for orchestration.
3.2 Agents (Workers)
Execution units for builds.
Types of Agents:
Static: Manually provisioned machines.
Dynamic: Auto-provisioned machines using cloud integrations (Docker/Kubernetes).
Connection methods:
SSH: Standard for Unix/Linux.
JNLP: Java Network Launch Protocol, often used where direct connection from the master is not possible.
3.3 Communication between Master and Agents
Master → Agent (via SSH): Issues commands.
Agent → Master (via JNLP): Agents connect to master if behind firewalls.
4. Jenkins Agents - Permanent and Cloud Based
4.1 Permanent Agents
Always online
Must be manually maintained
Suitable for dedicated build machines (e.g., hardware-dependent builds)
Setup:
Install Java
Configure SSH access
Connect using agent settings
4.2 Cloud-Based Agents
Spun up on demand
Destroyed after build completion
Great for ephemeral workloads, scaling, cost control
Plugins:
Kubernetes Plugin
Docker Plugin
EC2 Plugin
Azure VM Agents
Provisioning:
Templates define how agents are configured when created.
Labels are used to match jobs to specific agent templates.
5. FreeStyle Builds and Pipelines
5.1 FreeStyle Builds
Simple UI based jobs
Suitable for very simple workflows
Drag and drop configuration
Not reusable or modular
Example FreeStyle Steps:
Pull source from Git
Run shell script to build
Send email on failure
Limitations:
Difficult to manage at scale
No easy versioning
5.2 Pipelines
As Code (Groovy DSL)
Highly modular and maintainable
Easy to store and review with code
Supports parallelism, retries, parameters, approvals, etc.
Types of Pipelines:
Declarative Pipelines: Simplified, structured.
Scripted Pipelines: Powerful but complex.
6. Setting up Jenkins using Docker
6.1 Why Docker?
Easy installation
Consistent environment
Isolation
Simpler upgrades
6.2 Running Jenkins via Docker
bashCopyEditdocker pull jenkins/jenkins:lts
docker run -d \
--name jenkins \
-p 8080:8080 -p 50000:50000 \
-v jenkins_home:/var/jenkins_home \
jenkins/jenkins:lts
Important Options:
-v jenkins_home:/var/jenkins_home
→ Persistence across container restarts.
7. Jenkins Web GUI Walkthrough
7.1 Dashboard
Job List
Current Build Status
Build History
7.2 Manage Jenkins
Configure System
Plugin Management
Credentials Management
Security Settings
Global Tool Configuration
8. Creating a Simple FreeStyle Job
Steps:
"New Item" → "Freestyle Project"
Source Code Management: Git → Add Repo URL
Build Triggers: Poll SCM or GitHub webhook
Build Environment: Add shell script or batch commands
Post Build Actions: Archive artifacts, send notifications
Pro Tip: Always define a build number or build description for clarity.
9. Exploring The Jenkins Filesystem and Workspace
Important Paths:
Path | Purpose |
/var/jenkins_home/jobs/ | Stores job configurations and histories |
/var/jenkins_home/workspace/ | Active working copies of repositories |
/var/jenkins_home/plugins/ | Installed plugins |
/var/jenkins_home/secrets/ | Credentials and API tokens |
10. Freestyle job - running Python scripts with Jenkins
Example:
Install Python on Master/Agent.
Create a Freestyle job.
Build Step:
bashCopyEdit#!/bin/bash
python3 my_script.py
- Save and Build.
Important:
Use virtualenv for dependency isolation:
bashCopyEditpython3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
11-13. Setting up Docker Cloud Agents and Templates
Install and configure Docker plugin
Manage Plugins → Install "Docker Plugin"
Configure Docker cloud settings (Docker Host URI, credentials)
Define templates:
Base Docker image
Labels
Volumes
Commands to execute
14. Using labels to restrict Jobs to Agents
In Job Configuration:
Check "Restrict where this project can be run"
Enter Label (e.g.,
docker-agent
)
This ensures the job only runs on matching labeled agents.
15. Setting Builds to be automatically triggered on commits
GitHub Setup:
Settings → Webhooks → Add:
perlCopyEdithttp://<your-jenkins-server>/github-webhook/
Jenkins Job:
- Build Triggers → GitHub hook trigger for GITScm polling
16. Setting up Declarative Pipelines using Groovy
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'mvn clean install'
}
}
stage('Test') {
steps {
sh 'mvn test'
}
}
stage('Deploy') {
steps {
echo 'Deploying to Production'
}
}
}
}
Features:
Retry on failure
Parallel stages
Post actions (always, success, failure)
17. Using a Jenkinsfile for pipelines
Jenkinsfile Example
groovyCopyEditpipeline {
agent { label 'docker-agent' }
environment {
DOCKER_CREDENTIALS = credentials('dockerhub-creds')
}
stages {
stage('Checkout') {
steps {
git url: 'https://github.com/example/myrepo.git'
}
}
stage('Build') {
steps {
sh 'docker build -t myimage:latest .'
}
}
stage('Push') {
steps {
sh 'docker login -u $DOCKER_CREDENTIALS_USR -p $DOCKER_CREDENTIALS_PSW'
sh 'docker push myimage:latest'
}
}
}
}
18. Jenkins BlueOcean
Benefits:
Visualize Pipelines
Drag-and-drop editor
Easy configuration
GitHub integration
Setup:
Install BlueOcean plugins
Access
http://localhost:8080/blue
Best Used With:
Multibranch pipelines
GitHub organizations
Conclusion
Jenkins is the backbone of modern Continuous Integration/Continuous Deployment (CI/CD).
It evolved from a simple build server into a powerful, scalable automation engine that can:
Build and test applications,
Deploy across different environments,
Automate virtually any task related to software delivery.
Mastering Jenkins equips developers, testers, and DevOps engineers with the ability to build robust delivery pipelines, integrate various tools, reduce errors, and accelerate software releases.
By learning:
How Jenkins is structured (Master/Agents),
How to create jobs and pipelines,
How to scale Jenkins using Docker/Kubernetes,
How to manage Jenkins securely and efficiently,
you can fully automate your software lifecycle — from a simple "push to Git" to "live deployment on production" — with confidence and reliability.
Subscribe to my newsletter
Read articles from Arijit Das directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
