CI/CD pipeline for 2-Tier Flask Todo Web Application

Varun MargamVarun Margam
12 min read

Table of contents

📍Introduction

🚀Welcome to my Jenkins blog series, where we'll dive into crafting a declarative pipeline for a 2-tier Flask Todo Web Application with MySQL. Follow our step-by-step journey as we set up a holistic pipeline, covering cloning, building, DockerHub integration, and deployment. Let's get started!💡

To ensure a seamless experience, make sure to check out my previous blog, which will help you with an easy understanding and implementation of the steps outlined in this blog:📚

Introduction to Jenkins Pipelines: Streamlining Your Development Workflow

📍Pipeline Structure

📍Declarative Pipeline for 2-Tier Flask App

Step 1: Fork this GitHub repository to your GitHub account:

Repository URL: https://github.com/Varunmargam/2-tier-flask-todo-cicd

Step 2: Create an AWS EC2 instance with Ubuntu AMI, select t2.medium, and allow "HTTPS, and HTTP traffic from the internet". This is for the app, and the pipeline to run smoothly.

Step 3: Installation and setup of all the necessary services: You can skip if you have docker and Jenkins already installed

# Installing docker & docker-compose
sudo apt update
sudo apt install docker.io -y # -y flag means yes permission.
sudo apt install docker-compose -y
sudo systemctl enable docker

# Installing Java for Jenkins
sudo apt update
sudo apt install openjdk-17-jre
java -version # verify java installation

# Installing Jenkins
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins

# Allow permissions for jenkins user to execute docker commands
sudo usermod -aG docker jenkins # adding jenkins user into docker group.
sudo systemctl reboot # Or you can stop and start the instance.

The Setup for our project has been completed.

Step 4: Go to the "security" section of your EC2 instance, select the security group Id, and click on "Edit Inbound rules". Make sure you allow traffic from Anywhere IPv4 to ports 8080 (for Jenkins), 5000(for the Flask app), and 3306(for MySQL).

Click on "Save rules".

Step 5: Go to your browser and type public_IP_address of your aws instance**:8080** ec2__public_ip:8080 and click on "New Item".

Give the job name: "Flask-todo-app-cicd", select "Pipeline", and click "Ok". We will be making a declarative pipeline.

Step 6: Give a description:

Step 7: Check the "GitHub project" option and in the "Project url" field enter the URL for the GitHub repository which contains this Flask Web Application :

Copy the Forked GitHub repository URL:

Step 8: Click on the "Advanced" dropdown and add "Display name" as "Flask Todo Web App"

Step 9: Scroll down to the "Advanced Project Options" click on the "Advanced" dropdown and add the same "Display name" as "Flask Todo Web App"

Step 10: Come to the "Pipeline" section and add this pipeline script:

This pipeline script is to test the working of the pipeline ( optional )

pipeline{
    agent any

    stages{
        stage("Clone"){
            steps{
                echo "Code cloned"
            }
        }
        stage("Build"){
            steps{
                echo "Code built"
            }
        }
        stage("Push to Repository"){
            steps{
                echo "docker image pushed to the DockerHub repository"
            }
        }
        stage("Deploy"){
            steps{
                echo "Flask app has been deployed"
            }
        }
    }
}

Step 11: Click on Save, now our Pipeline has been created to test try and run this pipeline by clicking on "Build Now".

Our first build of the pipeline has been successful.

Now let's define the steps of the stages one by one.

Step 12: For the stage ("Clone") we will have to write the Groovy syntax to clone the Git repository from GitHub:

  1. Google "groovy syntax for git clone", and I got this result from StackOverflow:

  2. Copy the HTTP URL of the GitHub repository:

Type the groovy syntax for git clone:

stage("Clone"){
    steps{
        git url: 'https://github.com/Varunmargam/2-tier-flask-todo-cicd.git', branch: 'master'
    } // branch is to mention the branch we want to clone, here in our case 'master'
}

Step 13: Click on "Save" and start the build of the pipeline.

Instead of running the whole build process after we have defined all the stages. It is always a good practice to define the steps in 1 stage and check if the build process is running successfully or not. This helps in troubleshooting if the build fails.

You can see our #2 build has been successful, check the logs of the "Clone" stage by clicking on it:

It will display the logs of that particular stage:

You can see in the logs it has cloned the Git repository inside the /var/lib/jenkins/workspace/ directory:

You can see the Flask-todo-app-cicd is a Git repository with ls -a command. It has a .git file.

Step 14: Define the Build process, using the command docker build . -t flask-todo-app-cicd. To execute shell commands inside the Groovy syntax file use sh "shell command":

Save this and run the pipeline to check if the build stage is successful or not.

You can check the Build stage "Logs" or the "Console Output". Executing the docker images command to verify the built images:

Step 15: Define the "Push to Repository" stage, the steps to push this Docker image we have built in the "Build" stage to the DockerHub from the terminal are:

  1. Login to the DockerHub: docker login -u <user_name> -p <password>

    <user_name> and <password> are the credentials to log in to DockerHub.

  2. Rename the image in the format username/image_name: docker tag <image_name> username/<image_name>:latest "latest" represents the version of the image.

    Since the images in the DockerHub repository are named in the above format.

  3. Push the image to DockerHub: docker push <new_image_name>:latest

Step 16: To execute the "Login to the DockerHub" step via Jenkins, we have to first give the DockerHub credentials to Jenkins.

  • Go to Dashboard and click on "Manage Jenkins":

  • Go to the "Security" section and click on "Credentials":

  • Click on "(global)":

  • Click on "Add Credentials" at the top right:

  • Enter the Username and Password of your DockerHub Account:

    Enter ID as "DockerHub" and give Description as "These are DockerHub credentials."

    Your DockerHub credentials that contain the username and password for your DockerHub account will be identified using this ID by Jenkins.

  • Click "Create":

  • Your DockerHub credentials are created with the ID "DockerHub".

Step 17: Go to your "Flask-todo-app-cicd" pipeline and click on "Configure" to define the "Login to the DockerHub" step inside the "Push to Repository" stage.

/* The Groovy Syntax has a function called withCredentials() that passes
 the credentials are passed as arguments to the function and inject into the
 scope of this function. */
stage("Push to Repository"){
    steps{
        withCredentials([usernamePassword(credentialsId:"DockerHub",passwordVariable:"DockerHubpass",usernameVariable:"DockerHubUser")]){
            sh "docker login -u ${env.DockerHubUser} -p ${env.DockerHubPass}"
        }
    }
}

  • withCredentials(): Groovy function that accepts credentials as arguments to be used inside the scope of its function.

    Inside this withCredentials() function we are passing a list of credentials mentioned in [].

  • usernamePassword(): This is a Groovy function used to pass the credentials.

  • credentialsId:"DockerHub": Key value pair.

  • passwordVariable:"DockerHubpass": Here we are defining a variable for the password with the name "DockerHubpass" same is for this usernameVariable:"DockerHubUser".

  • sh "docker login -u ${env.DockerHubUser} -p ${env.DockerHubPass}: Here we are accessing the credentials passed as arguments to this function, env here is the environment variable.

Step 18: Enter these steps, "Save" these changes, and start the "Build" process. This is to ensure that Jenkins can successfully log in to DockerHub.

You can see that the build has been successful. Check the "Console Output" of this #4 build.

As you see in the above "Console Output" "Login Succeeded".

Step 19: Let's start defining the "Rename the image in the format username/image_name" step and the "Push the image to DockerHub" step.

stage("Push to Repository"){
    steps{
        withCredentials([usernamePassword(credentialsId:"DockerHub",passwordVariable:"DockerHubpass",usernameVariable:"DockerHubUser")]){
            sh "docker login -u ${env.DockerHubUser} -p ${env.DockerHubPass}"
            sh "docker tag flask-todo-app-cicd ${env.DockerHubUser}/flask-todo-app-cicd:latest"
            sh "docker push ${env.DockerHubUser}/flask-todo-app-cicd:latest"
        }
    }
}

Step 20: "Save" this and click on "Build Now".

As you can see the #5 build has been successful now let's see the "Console Output" and see if the Docker image has been pushed to DockerHub or not.

You can see the image flask-todo-app-cicd has been pushed to my DockerHub account.

Now my original docker-compose.yml was this:

version: '3'
services:

  backend:
    build:
      context: .
    ports:
      - "5000:5000"
    environment:
      MYSQL_HOST: mysql
      MYSQL_USER: root
      MYSQL_PASSWORD: root@123
      MYSQL_DB: todo_db  # Create a new database for the to-do app
    depends_on:
      - mysql

  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: root@123
      MYSQL_USER: varun
      MYSQL_PASSWORD: varun@123
    volumes:
      - mysql-data:/var/lib/mysql
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql

volumes:
  mysql-data:

Here, the backend service is building the image from the contents of my project. But I want to use the image that we have pushed to the DockerHub. For that, I have updated my docker-compose.yml file to this:

version: '3'
services:

  backend:
    image: "varunmargam/flask-todo-app-cicd:latest"  # Use the image from DockerHub
    ports:
      - "5000:5000"
    depends_on:
      - mysql

  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: root@123
      MYSQL_USER: varun
      MYSQL_PASSWORD: varun@123
    volumes:
      - mysql-data:/var/lib/mysql
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql

volumes:
  mysql-data:

Step 21: Define the "Deploy" stage using these commands:

stage("Deploy"){
    steps{
        sh "docker-compose down && docker-compose up -d"
    }
}

While building the pipeline multiple times we have to stop the running containers and remove these stopped containers from the previous build. To achieve this we first execute docker-compose down and then execute docker-compose up -d.

Step 22: "Save" this and build the pipeline:

As you can see our #6 build has been successful, which means our application has been successfully deployed.

Go to your browser and type this URL: ec2_public_ip:5000

You can add some tasks to the list:

And can click on checkboxes to remove the completed task:

Let's also add Webhooks to the pipeline so that we can achieve Continuous Deployment i.e. it will automatically trigger the Build pipeline as soon as changes are committed to the code. Thereby achieving "Continuous Integration and Continuous Deployment".

Step 23: Go to your Forked GitHub repository, click on "Settings", and click on "WebHooks" at the left:

Step 24: Add the Jenkins Web Interface URL in the "Payload URL" field in the format jenkins_web_url/github-webhook/ as you can see in the above image. Click on "Send me everything" in the events section and click on "Add webhook".

You will see your webhook has been created and is active if it shows a "green tick" mark:

If you do not see the tick refresh the page.

Step 25: Go to the Jenkins Web Interface Dashboard, click on "Configure", go to the "Build Triggers" section, and check on "GitHub hook trigger for GITScm polling".

Click on "Save", and adjust your GitHub tab and the Jenkins tab side by side like this:

Step 26: Make some changes to my code and commit them to verify whether the build of the pipeline gets automatically triggered or not.

Go to the index.html file inside the templates folder and click on "edit" which will be at the left corner represented by a "pencil".

I will be making changes in the <h1> tag so that the changes are easily visible:

Originally the <h1> tag contained a "To-Do List"

The change I made was:

And I committed those changes:

You can see the build process has been started as soon as I committed those changes

The build process has been successful. Let's see if the changes have been reflected in the updated app:

Yay!!🥳 The changes have been successfully implemented, indicating that our pipeline setup aligns with our intended configuration.

( Apology for the spelling mistake😅)

You can also trigger this CICD pipeline without using the Jenkins Web Interface by adding the pipeline script inside the file named Jenkinsfile along with your project code in the GitHub repository.

This way whenever you will commit the changes to the code in GitHub the Webhook will trigger the Jenkins build of the Jenkins pipeline and when Jenkins will locate and read the Jenksfile it will execute the stages as defined in the Jenkinsfile.

( My repository code already contains the Jenkinsfile so you can skip to the step )

To upload the Jenkinsfile to your GitHub repo follow these steps:

Step 1: Go to your GitHub repository, click on "Add file", and click "Create new file"

Step 2: Go to your Jenkins Web interface, click on "Configure", copy the "Pipeline script", and paste it into this GitHub file.

pipeline{
    agent any

    stages{
        stage("Clone"){
            steps{
               git url: 'https://github.com/Varunmargam/2-tier-flask-todo-cicd.git', branch: 'master' 
            }
        }
        stage("Build"){
            steps{
                sh "docker build . -t flask-todo-app-cicd"
            }
        }
        stage("Push to Repository"){
            steps{
                withCredentials([usernamePassword(credentialsId:"DockerHub",passwordVariable:"DockerHubPass",usernameVariable:"DockerHubUser")]){
                    sh "docker login -u ${env.DockerHubUser} -p ${env.DockerHubPass}"
                    sh "docker tag flask-todo-app-cicd ${env.DockerHubUser}/flask-todo-app-cicd:latest"
                    sh "docker push ${env.DockerHubUser}/flask-todo-app-cicd:latest"
                }
            }
        }
        stage("Deploy"){
            steps{
                sh "docker-compose down && docker-compose up -d"
            }
        }
        stage("Remove unused images"){
            steps{
                sh "docker image prune -af"
            }
        }
    }
}

Name the file "Jenksfile" at the top:

Do not commit the changes now

Step 3: Go to your Jenkins Web Interface, go to your pipeline, click on "Configure", scroll down to the "Pipeline" section, delete the "Pipeline Script", in the Definition field click on the dropdown, and select "Pipeline script from SCM". SCM - Source Code Management.

  • In the "SCM" section select "Git".

  • In the "Repository" section add the HTTP URL inside the "Repository URL" field.

  • Scroll down to the "Branches to build" section and type "*/master" and click "Save"

Step 4: Again adjust the two windows like this:

Step 5: Commit the changes:

You can see the build process has been triggered via webhook as soon as the changes were committed:

But you can see that an additional stage has been added at the beginning of "Declarative Checkout SCM" It means that Jenkins went to GitHub and did Declarative checkout i.e. located the Jenkinsfile and then started the build process of the pipeline as defined in the Jenkinsfile.

Now if Jenkins keeps on building new containers, the images run by previous containers will be remained unused in the system and will occupy space. This may cause the memory to fill in the pipeline and keeps executing after a certain number of times.

Let's add another stage in the Pipeline that removes the unused images using the command:

docker image prune -af

I went to my GitHub repository and edited Jenkisfile by adding the "Remove unused images" stage.

And committed to those changes:

This commit will trigger the pipeline build:

You can see now a new stage has been added at the end that removes the unused images from the system:

The images created due to the above build:

Again built the pipeline:

Now you can see the previous images were removed and only the images created in the above build exist in the system:

Congratulations!!🥳 We have successfully created a Declarative pipeline for a 2-tier Flask Todo Web Application.🎉

📍Conclusion

As we conclude this blog, you've not only gained insights into creating a robust pipeline but also learned how to streamline the process with Git integration and SCM synchronization. Your journey toward mastering Jenkins automation has just begun. Keep on practicing and craft more pipelines for different applications.🚀🌟

Thank you for reading this blog! 📖 Hope you have gained some value. In the future blog, we will be creating and exploring Jenkins pipelines.

If you enjoyed this blog and found it helpful, please give it a like 👍, share it with your friends, share your thoughts, and give me some valuable feedback.😇 Don't forget to follow me for more such blogs! 🌟

📍Reference

5
Subscribe to my newsletter

Read articles from Varun Margam directly inside your inbox. Subscribe to the newsletter, and don't miss out.

#90daysofdevopsJenkinsCI/CDDevOps Journey#day27

Written by

Varun Margam
Varun Margam