Streamline Your DevOps Workflow: Connect Jenkins and Azure DevOps

I'm excited to continue my DevOps learning journey and resume the "MyDevOpsSpace" series after months later. During this time, I’ve had the opportunity to step into a new role, which has provided a valuable shift in my focus.

I’m thrilled to take the next step in my DevOps path as I began my role as a Level 1 Server Engineer. This position not only offers me practical, hands-on industry experience but also deepens my understanding of the key technologies relevant to my work. I’m eager to apply what I've learned and continue growing in this field.

I decided to do more activities using Jenkins since it is widely used across industries. By Resuming my first blog: Jenkins Setup on VM: My First Flight

AIM

I revisited and practiced it to re-acquire the concepts related to it. Now, I am deciding to do more by running CICD automation by syncing Azure DevOps with it.

For this, I had to take different approach.

I decided to configure a new VM in Azure which consisted of below hardware configuration & installed Jenkins in it with required supported packages:

• Publisher: Canonical

• Offer: Ubuntu Server

• SKU: 22.04-LTS

• Version: Latest

REASON: In the Azure VM with OS image as "Bitnami package for Jenkins", I was facing major hurdles.

1. Demand for latest version for VSTS plugin i.e. Azure DevOps since my pipeline run fails accordingly, throwing the error similar to it.

2. Having limited functionalities.

3. My ssh connection gets timed out after certain minutes! For which I had to re-login to Jenkins VM during crucial moments.

NOTE: Can be configured in AWS/GCP/On-Premises as well.

I attached NSG in VM subnet by allowing ports within Inbound Rule - HTTP (Port:80 & Port:8080), HTTPS (Port:443), SSH (Port:22).

Talking about VM size, I relied on Standard D2S-v2 to ensure smooth run. Use at least Standard_B1ms (1 vCPU, 2GB RAM) for basic usage.

Last but not the least, I attached Public IP to VM NIC card to access VM through ssh connection using public/private RSA SSH key in .pem file format.

As for Jenkins, it requires these below resources stepwise within LINUX VM (Debian/RHEL),

1. Install Java

2. Add Jenkins Repository

3. Adjust Firewall (Port:8080)

4. Install Jenkins

5. Access Jenkins

Once VM is configured, I accessed it through ssh port:22 using public IP address, making sure I access the private key file .pem within file directory.

ssh -i <private_key_file>.pem azureuser@<vm-public-ip>

NOTE: Its good practice to keep packages up to date by running below command every time interval if any major change/update has been done.

sudo apt update

OR

sudo apt update && sudo apt upgrade -y

Install Java

Jenkins requires Java. Install OpenJDK 11 or 17 (both are supported). I used Java OpenJDK 17 for my below

sudo apt install openjdk-17-jdk -y

Verify installation:

java -version

Add Jenkins Repository

Add the official Jenkins repository and key:

curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
/usr/share/keyrings/jenkins-keyring.asc > /dev/null

echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null

Adjust Firewall (Port:8080)

Although we enabled http - port:8080 in inbound rule of NSG. If using UFW (Uncomplicated Firewall), allow Jenkins through port 8080:

sudo ufw allow 8080
sudo ufw reload

Install Jenkins

Now that most of the prerequisite for Jenkins (like Java, Repositories etc.) has been setup, update the packages:

sudo apt update

Jenkins Install:

sudo apt install jenkins -y

Access Jenkins

Start Jenkins serial wise as provided below and ensure it starts on boot:

1. sudo systemctl start jenkins
2. sudo systemctl enable jenkins
3. sudo systemctl status jenkins

##To STOP Jenkins:
sudo systemctl stop jenkins

##To RESTART Jenkins:
sudo systemctl restart jenkins

On running sudo systemctl status jenkins, it’s important that the state of Jenkins be active (running).

ssh connection is taken to Jenkins VM via Public IP (format mentioned above). I received the home page of Jenkins.

To fetch the credentials for accessing Jenkins (Username & Password) as below, it’s present in /var/lib/jenkins/secrets/initialAdminPassword file. make sure to access the file using admin sudo privileges.

sudo cat /var/lib/jenkins/secrets/initialAdminPassword 
##On running we get long mixed character password

NOTE: It’s important to note the differences among,

sudo systemctl start jenkins 
sudo systemctl enable jenkins 
sudo systemctl status jenkins 
sudo systemctl stop jenkins 
sudo systemctl disable Jenkins

sudo systemctl start jenkins

• Purpose: Starts the Jenkins service immediately.

• Effect: Jenkins will run now, but it will not start automatically after a reboot.

sudo systemctl enable jenkins

• Purpose: Enables Jenkins to auto-start on system boot.

• Effect: Jenkins will start automatically after every system reboot.

sudo systemctl status jenkins

• Purpose: Shows the current status of Jenkins.

• Effect: Displays whether Jenkins is active (running), inactive (stopped), or if there are any errors.

sudo systemctl stop jenkins

• Purpose: Stops the Jenkins service immediately.

• Effect: Jenkins will stop running, but it’s not disabled.

• After Reboot: If Jenkins was enabled, it will start again after a reboot.

• Use Case:

  • Use this when you want to temporarily stop Jenkins.

  • It preserves your data, configurations, and jobs.

  • You can restart it anytime with sudo systemctl start jenkins.

sudo systemctl disable jenkins

• Purpose: Disables Jenkins from starting automatically at boot.

• Effect: Jenkins will not start after a system reboot.

• Important:

  • Does NOT delete any data, configurations, or installed packages.

  • You can still manually start Jenkins with sudo systemctl start jenkins.

All these setup for Jenkins were not easy. I faced major challenges which made me understand & resolve the Jenkins to its core!

CHALLENGES

After setting up Jenkins following above steps, I kept getting this below error,

My Jenkins fails to startup after configuration

On diagnosis (finding it on websites), I realized there could be two possible reasons:

1. Incompatible JAVA version for running Jenkins.

2. /lib/systemd/system/jenkins.service file which is responsible to activate Jenkins (as highlighted above) within VM may not be correctly configured.

TROUBLESHOOTED

First, it’s better to cross check always that all the LINUX packages are installed & up to date. If all looks good then proceed accordingly,

• ATTEMPT 1: JAVA versions

  Make sure you are using OpenJDK 11 or 17, as Jenkins is compatible with these versions. Check the version using

    java -version
  

  If the version is incompatible, install the correct version:

•             sudo apt install openjdk-11-jdk-y
              ##OR##
              sudo apt install openjdk-17-jdk -y
  

  Configure Java Environment Variables

  Set up environment variables for Java.

•             ##Edit the environment file:##
              sudo vim /etc/environment
  
              ##Add the Java path:##
              JAVA_HOME="/usr/local/java"
              PATH="$JAVA_HOME/bin:$PATH"
  
              ##Reload the environment variables:##
              source /etc/environment
  
              ##Verify the JAVA_HOME variable:##
              echo $JAVA_HOME
  

  Restart Jenkins sudo systemctl restart jenkins & Access it again using the format - http://<your-server-ip>:8080!

• ATTEMPT 2: Set “jenkins.service” File -

  /lib/systemd/system/jenkins.service file correctly

  First, it’s important to keep an update what inputs/variables are set within jenkins.service file.

  The file needs to contain these variables:

  1. Java & Jenkins Environment

  2. ExecStart - Execution Start containing /usr/share/jenkins/jenkins.war.

  3. Restart status with time period

  4. Jenkins with correct user and group permissions

  5. WorkingDirectory - Working Directory of Jenkins data.

Updated /lib/systemd/system/jenkins.service File

•           [Unit]
            Description=Jenkins Continuous Integration Server
            Documentation=https://www.jenkins.io/doc/
            After=network.target
  
            [Service]
            Environment="JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"  # Ensure this matches your installed Java version                                                          
            ExecStart=/usr/bin/java -Djenkins.install.runSetupWizard=false -jar /usr/share/jenkins/jenkins.war
            User=jenkins
            Group=jenkins
            WorkingDirectory=/var/lib/jenkins
            Restart=always
            RestartSec=5s
  
            [Install]
            WantedBy=multi-user.target
  

  Steps to Apply the Changes:

  1. Save the File: After editing the file, save and exit (e.g., using nano):

      sudo vim /lib/systemd/system/jenkins.service
     

  2. Reload Systemd:

      sudo systemctl daemon-reload
     

  3. Restart Jenkins Service:

      sudo systemctl restart jenkins
      sudo systemctl status jenkins
     

  4. Verify Logs if Issues Persist:

      sudo journalctl -u jenkins -xe
      sudo tail -f /var/log/jenkins/jenkins.log
     

• ATTEMPT 3: Missing File -

  /usr/share/jenkins/jenkins.war used in /lib/systemd/system/jenkins.service file

  It’s linked to above issue (ATTEMPT 2) while resolving the issue of jenkins.service file. My jenkins.war was not present within its respective directory as called within ExecStart variable, which was a bit hectic to resolve it, since it was necessary to execute jenkins.service service to run Jenkins.

  First, I verified its location by executing

    ls -ltr /usr/share/jenkins/jenkins.war
  

  Install jenkins.war If Missing:

    sudo apt-get install --reinstall jenkins
    ##OR##
    wget -O /usr/share/jenkins/jenkins.war https://get.jenkins.io/war-stable/latest/jenkins.war
  

  Once installed, re-check the jenkins.war file iis present through ls -ltr as above.

• After ensuring the correct JAVA version is installed, execute the jenkins.service file and installed Jenkins as guided above.

  NOTE: Keep in mind, if any issues encountered, Always check the logs:

    sudo tail -f /var/log/jenkins/jenkins.log
  

MY PLAN

After successful Jenkins, I imported a GitHub Repository from online which contained JAVA based Maven project which will execute & setup this below .jar file in my tomcat server

spring-boot-initial-0.0.1-SNAPSHOT.jar

DISCLAIMER: Used same Repository in my previous JENKINS blog as well

ARTIFACTS:

GITHUB Link:

https://github.com/prak233/springBoot233.git

AZURE REPO Link:

https://SMG233@dev.azure.com/SMG233/JavaSpace/_git/JavaSpace

NOTE: For this project I used both Azure Repo & GitHub Repo for multiple pipeline execution RUNS and observing the OUTPUTS generated. For which there are some branches being cloned with for experimentation.

Since I used Azure Repo, I also relied on Azure DevOps. By integration of my Jenkins with Azure DevOps with the help of Service Hook (OR) Service Connection as explained in my previous blog.

After execution of BUILD artifact within Jenkins, I setup my CI/CD YAML scripts in Azure DevOps where the build part is pulled/downloaded from Jenkins BUILD artifact & simultaneously executed to tomcat server.

Once YAML Pipeline is configured, I placed my YAML script.

MY YAML SCRIPT:

trigger:
- none

pool:
  vmImage: windows-latest

stages:
##Stage 1: Execution of BUILD job
- stage: BuildStage
  displayName: BuildStage
  jobs:
  - job: CIJob
    displayName: "Jenkins Build JOB"
    steps:
      ###Performing JENKINS Queue Job
    - task: JenkinsQueueJob@2
      displayName: "Queue Task"
      inputs:
        serverEndpoint: 'Jenkins-SRC'
        jobName: 'revisit-CICD'
        isMultibranchJob: true
        multibranchPipelineBranch: 'feature/test'
        captureConsole: true
        capturePipeline: true
        isParameterizedJob: false

      ###Jenkins DOWNLOAD Job
    - task: JenkinsDownloadArtifacts@2
      displayName: "Download Artifact Task"
      inputs:
        jenkinsServerConnection: 'Jenkins-SRC'
        jobName: 'revisit-CICD'
        saveTo: '$(Build.StagingDirectory)'
        jenkinsBuild: 'LastSuccessfulBuild'

        ###PUBLISHING Build Artifact  
    - task: PublishBuildArtifacts@1
      displayName: "Publish Artifact Task"
      inputs:
        PathtoPublish: '$(Build.StagingDirectory)'
        ArtifactName: 'drop'
        publishLocation: 'Container'


##Stage 2: Execution of DEPLOY Job
- stage: DeployStage
  dependsOn: BuildStage
  displayName: DeployStage
  jobs:
  - deployment: deploytoVM
    displayName: 'deploytoVM'
    environment:
      name: 'MyEnv'
      resourceType: VirtualMachine
      tags: 'linux'
    strategy:
      runOnce:
        deploy:
          steps:
            ###Converting the path from BUILD artifact to PIPELINE
          - task: DownloadPipelineArtifact@2
            displayName: "Download Pipeline Build Artifact"
            inputs:
              artifactName: drop
              targetPath: '$(pipeline.artifact)'

            ###Copy Files to Linux VM
          - task: AzureCLI@2
            displayName: 'Copy Files to Linux VM'
            inputs:
              azureSubscription: 'Pay-As-You-Go(52ebb814-d1d8-427b-badb-92e9e2108392)'
              scriptType: 'bash'
              scriptLocation: 'inlineScript'
              inlineScript: |                ####USING LINUX scripting####
                # Define variables

                ##Update the ARTIFACT_PATH variable in your pipeline:
                ARTIFACT_PATH="/azagent/_work/1/drop/initial/target/spring-boot-initial-0.0.1-SNAPSHOT.jar"

                VM_IP='52.229.163.139'
                USERNAME='pratul'
                SSH_KEY_PATH='/azagent/revisit-linuxVM_key.pem'
                ls -l $SSH_KEY_PATH
                ls -l $ARTIFACT_PATH

                ##Ensure the deploy directory exists on the target VM
                mkdir -p /home/$USERNAME/deploy/

                # Copy JAR file to the Linux VM
                scp -i $SSH_KEY_PATH -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $ARTIFACT_PATH $USERNAME@$VM_IP:/home/$USERNAME/deploy/

            ###Install Java & Start Spring Boot Application
          - task: AzureCLI@2
            displayName: 'Install Java & Start Spring Boot Application'
            inputs:
              azureSubscription: 'Pay-As-You-Go(52ebb814-d1d8-427b-badb-92e9e2108392)'
              scriptType: 'bash'
              scriptLocation: 'inlineScript'
              inlineScript: |                ####USING LINUX scripting####
                VM_IP='52.229.163.139'
                USERNAME='pratul'
                SSH_KEY_PATH='/azagent/revisit-linuxVM_key.pem'

                ssh -i $SSH_KEY_PATH -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $USERNAME@$VM_IP << 'EOF'
                set -ex

                DEPLOY_DIR="/home/$USERNAME/deploy"
                JAR_FILE="spring-boot-initial-0.0.1-SNAPSHOT.jar"

                # Kill any existing process on port 8080
                sudo fuser -k 8080/tcp || true

                # Start the application and log output
                nohup java -Xmx512m -Xms256m -jar \$DEPLOY_DIR/\$JAR_FILE > \$DEPLOY_DIR/app.log 2>&1 &

                sleep 5

                # Check if the process started
                if pgrep -f "\$JAR_FILE"; then
                    echo "✅ Spring Boot application started successfully!"
                else
                    echo "❌ ERROR Spring Boot application failed to start."
                    echo "📄 App Log:"
                    cat \$DEPLOY_DIR/app.log
                    exit 1
                fi

                EOF

AIM

1. Downloads the Spring Boot JAR file from the pipeline artifact storage.

2. Uses SCP to copy the JAR file to /home/azureuser/deploy/ on the Linux VM.

3. SSH into the Linux VM and:

   • Kills any existing process running on port 8080.

   • Starts the Spring Boot application in the background.

   • Logs output to app.log for debugging.

   • Verifies the process is running.

4. Ensure Java is installed on your Linux VM (sudo apt install openjdk-11-jdk if needed)

   OR

   Deploy through YAML

   VERIFICATION

5. Run ps aux | grep spring-boot-initial on your Linux VM to check if the process is running.

6. Access the application at http://VM_PUBLIC_IP:8080.

Challenges:

Facing error in "Copy Spring Boot JAR to Linux VM" task

Attempt:

1. Verify if Azure CLI is Installed

   Run the following command in your agent machine (or use a script task in the pipeline):

az --version

If it's not found, proceed to install it.

2. Install Azure CLI on Your Self-Hosted Agent

For Ubuntu-based Linux (including Azure VM):

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

For Windows:

Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi
Start-Process msiexec.exe -ArgumentList "/i AzureCLI.msi /quiet" -Wait

📌 3. Restart the Agent

After installing the CLI, restart your agent service.

After this, On Execution of Pipeline, I faced an another issue:

Challenges:

Unable to fetch .pem key file required for ssh connection

Attempt:

Upon checking I found .pem key file is missing in LINUX VM (tomcat).

1. I moved it to my LINUX VM from local PC using scp (Secure Copy Protocol).

   METHOD 1: Using scp

   • Open Command Prompt (cmd) or PowerShell on your local PC.

   • Run the following SCP command (replace placeholders with your actual values):

       scp -i "<file-path of local PC>" "<file-path of local PC>" azureuser@VM_IP:/home/azureuser/
       #Eg
       scp -i "C:\Users\Pratul\Downloads\revisit-linuxVM_key.pem" "C:\Users\Pratul\Downloads\revisit-linuxVM_key.pem" azureuser@VM_IP:/home/azureuser/
     

     🔹 Explanation:

     • scp → Secure copy command.

     • -i "<file-path of local PC>" → Specifies the private key for authentication.

     • "C:\Users\Pratul\Downloads\revisit-linuxVM_key.pem" → Path to the .pem file on your local machine.

     • azureuser@VM_IP:/home/azureuser/ → The target VM location:

       • azureuser → Your VM’s username.

       • VM_IP → Your Azure VM’s public IP address.

       • /home/azureuser/ → Destination directory on the VM.

   • Verify the file upload: SSH into the VM and check if the file exists:

       bashCopyEditls -l /home/azureuser/revisit-linuxVM_key.pem
     

METHOD 1: Using WinSCP (GUI-Based Tool)

• If you prefer a graphical interface, WinSCP makes it easy.

  1. Download and install WinSCP from here.

  2. Launch WinSCP and enter:

     • File Protocol: SFTP

     • Host Name: Your VM’s public IP.

     • Username: Your VM’s username (e.g., azureuser).

     • Private Key File: Load your .pem file (convert it to .ppk using PuTTYgen if needed).

  3. Click "Login" → Drag and drop the .pem file into /home/azureuser/.

2. Once .pem is moved, make sure to set correct permissions:

    chmod 600 /home/azureuser/revisit-linuxVM_key.pem
    ##600 defines,
    ##6 -> read + write at USERS level
    ##0 -> read at both GROUPS & OTHERS levels
   

   But still issue occurs with same error

   REASON: it has -rw-r--r--, SSH will reject the key.

   Ensure it has permissions as -rw------- outputs:

    -rw------- 1 pratul pratul 1708 Jan 29 12:00 /azagent/revisit-linuxVM_key.pem
   

   RUN (if necessary):

    chmod 400 /azagent/revisit-linuxVM_key.pem
   

3. Check If Azure DevOps Agent Has Access to the Private Key

   Azure DevOps pipeline agent must have read access to <key-file>.pem.

   Run:

    sudo chown azureagentuser:azureagentuser /azagent/revisit-linuxVM_key.pem
   

   🔹 Replace azureagentuser with the actual Azure DevOps agent user.

This resolves my key issue.

Upon execution of pipeline, there is failure in HOST KEY verfification.

Challenges:

Error even though correct path is given with correct permission as -rw at user level.

Attempt:

# Debugging: Show file permissions
      ls -l $SSH_KEY_PATH
      ls -l $ARTIFACT_PATH

# Disable host key verification & Copy file securely
scp -i $SSH_KEY_PATH -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $ARTIFACT_PATH $USERNAME@$VM_IP:/home/$USERNAME/deploy/

🛠️ Explanation of Fixes

• -o StrictHostKeyChecking=no → Prevents SSH from prompting for host key acceptance.

• -o UserKnownHostsFile=/dev/null → Prevents SSH from writing host keys to ~/.ssh/known_hosts, avoiding host mismatch errors.

If the issue persists:

1. Manually test SSH from the Azure DevOps agent:

    ssh -i ~/.ssh/revisit-linuxVM_key.pem -o StrictHostKeyChecking=no pratul@52.175.37.163
   

   If SSH fails manually, it’s a VM, NSG, or firewall issue.

2. Ensure NSG allows SSH (port 22) inbound:

    az network nsg rule list --resource-group <ResourceGroup> --nsg-name <NSGName> --output table
   

3. Check SSH service on the VM:

    sudo systemctl restart sshd
    sudo systemctl status sshd
   

NOTE: The -p flag in the mkdir command stands for "parents", and it ensures that:

1. The directory is created only if it does not already exist.

2. Any missing parent directories are also created automatically.

This Mini PROJECT helped me understanding & using scripting of LINUX commands in more efficient way at proper place & time.

On researching, I found to learn new commands and scripts as below:

1. ps aux

   SSH into the VM and verify that the app is running:

    ssh -i /azagent/revisit-linuxVM_key.pem pratul@52.229.163.139
    ps aux | grep spring-boot-initial | grep -v grep
   

   👉 If the application is running, you should see an output with the java -jar command and its process ID.

2. netstat/ss -tulnp

   Check If the Application Is Listening on Port 8080

   Run this command inside the VM:

    sudo netstat -tulnp | grep 8080
   

   or

    sudo ss -tulnp | grep 8080
   

   👉 You should see an output like this if the app is running:

    tcp   LISTEN   0   50   0.0.0.0:8080   0.0.0.0:*   users:(("java",pid=12345,fd=XX))
   

   If you don’t see any output, the application is not running.

3. curl

   Check the Application Using CURL

   From your local machine, run:

    curl http://52.229.163.139:8080
   

   or, if your app has a health check endpoint:

    curl http://52.229.163.139:8080/actuator/health
   

   👉 If the app is working, you should see a response like:

    {"status":"UP"}
   

   If not, check firewall rules or application logs.

4. ufw

   Check firewall rules:

    sudo ufw status
   

   If port 8080 is blocked, open it:

    sudo ufw allow 8080/tcp
    sudo ufw reload
   

   For iptables, run:

    sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
   

Challenge:

Facing this issue even though my all prerequisites needed to install Spring Boot App.

Unfortunately, I debugged a lot on this & still working on this! I have also posted it on various social media to get solution for this error. It will be really Grateful if anyone contributes on my work🙏!

At last, I would like to conclude that even though my desired output is yet to come or not! I had a great fun in trying it out. Also gear up for more CI/CD automations works/projects which I will be trying out and experimenting to build my skills & knowlegde according by broading my scope of learning always!!😊.

REFERENCES:

• https://chatgpt.com/

• https://community.jenkins.io/c/using-jenkins/

• Course: Complete Linux Training Course to Get Your Dream IT Job 2025 | Udemy

• Best Azure and Cloud Technology Training | Bestitcourses

• https://youtube.com/