Phase 1 of Project: Infrastructure Setup

We need to set up the following components:

1. Jenkins

Install Jenkins for CI/CD pipeline management.
Install required CLI tools to interact with the Kubernetes cluster.
Install kubectl for Kubernetes operations.
Install Trivy, as Jenkins does not have it by default.
(Trivy is used to detect known security issues, such as CVEs — Common Vulnerabilities and Exposures — in applications and infrastructure before deployment*.)*
Install Java, which is a prerequisite for setting up Jenkins.
Install Docker to enable Jenkins to build and manage containerized applications.

2. Nexus

Set up Nexus Repository Manager using a Docker container.
Allocate a minimum of 4GB storage and 4GB RAM for smooth performance.

3. SonarQube

Deploy SonarQube Community Edition using Docker for code quality and static analysis.

4. Infrastructure Server (InfraServer)

Configure a dedicated VM to execute commands for creating and managing the EKS Cluster.
Install and configure the following tools on the InfraServer:
- AWS CLI – For managing AWS services.
- Helm – For Kubernetes package management.
- eksctl – For creating and managing EKS clusters.
- Terraform – For infrastructure as code (IaC) automation.

Prepare Infra VM

Create a VM for infrastructure setup and install required tools:

Install aws cli

sudo apt install -y unzip curl
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
Verify:

aws --version

Install Terraform

sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install terraform -y
terraform -version

Clone the Terraform repo and create EKS infra:

git clone <your_repo_url>
cd terraform-eks
terraform init
terraform apply -auto-approve

Phase 2: Kubernetes Configuration

Configure kubectl

curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin
kubectl version --short --client

If you run kubectl get nodes why there is an error

Eventhough cluster has been created but we are not connected to the cluster. i.e. we don’t have authentication so we need file kubeconfig file

Update kubeconfig:

aws eks --region ap-south-1 update-kubeconfig --name abhiproject-cluster

Install eksctl & Helm

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

IAM OIDC & Service Account

eksctl utils associate-iam-oidc-provider --region ap-south-1 --cluster abhiproject-cluster --approve

Why we need this command?

We want, inside kubernetes cluster the service account we need, should have permission to assume IAM role and should be able to create aws services

Create service account

eksctl create iamserviceaccount \
--region ap-south-1 \
--name ebs-csi-controller-sa \
--namespace kube-system \
--cluster abhiproject-cluster \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--approve \
--override-existing-serviceaccounts

Note: this command will use cloudformation in the baground to create service account.
This is needed to create volume and attach the volume to all worker node needed

These pods are needed to create ebs volume

Phase 3: CI/CD Tools Setup

Install Jenkins

Install Java:

sudo apt update
sudo apt install openjdk-11-jdk -y
sudo usermod -aG docker jenkins

Install LTS version of jenkins https://www.jenkins.io/doc/book/installing/linux/#debianubuntu

Install Trivy

sudo apt-get install wget apt-transport-https gnupg lsb-release

wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -

echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list

sudo apt-get update

sudo apt-get install trivy

Setup Nexus & SonarQube

Nexus:

For Nexus:

Install docker

And give permission or add docker in ubuntu group

sudo usermod -aG docker $USER

newgrp docker

sudo docker run -d --name nexus -p 8081:8081 sonatype/nexus3

SonarQube:
For SonarQube:
Install docker
And give permission or add docker in ubuntu group
sudo usermod -aG docker $USER
newgrp docker

docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Phase 4: Jenkins Integration

Install required Jenkins plugins: SonarQube Scanner, Docker, Kubernetes, Generic Webhook Trigger

Configure SonarQube under Manage Jenkins → Configure System

Configure Nexus credentials via Jenkins settings.xml

The artifact which created should be in maven releases or maven snapshots depending on env it is

For prod -- releases

For lower env -- snapshot

If jenkins wants to publish something to nexus. we need to made some changes in pom.xml file

Under settings.xml which in jenkins (manage file), make some changes. Add credentials to access the website.

note: manage file in jenkins occur only when we install config map provider plugin

Phase 5: Jenkins Pipeline

Create jenkins pipeline file For CI

pipeline {
    agent any

    tools {
        maven 'maven3'
    }

    environment {
        SCANNER_HOME = tool 'sonar-scanner'
        IMAGE_TAG = "v${BUILD_NUMBER}"
    }

    stages {
        stage('Git Checkout') {
            steps {
                git branch: 'main', credentialsId: 'git', url: 'https://github.com/abhi21993/Mega-Project_ci.git'
            }
        }

        stage('Compile') {
            steps {
                sh "mvn compile"
            }
        }

        stage('Testing') {
            steps {
                sh "mvn test"
            }
        }

        stage('Trivy FS Scan') {
            steps {
                sh "trivy fs --format table -o fs-report.html ."
            }
        }

        stage('Sonar Analysis') {
            steps {
                withSonarQubeEnv('sonar') {
                    sh '''
                        $SCANNER_HOME/bin/sonar-scanner \
                        -Dsonar.projectKey=gcbank \
                        -Dsonar.projectName=gcbank \
                        -Dsonar.java.binaries=target
                    '''
                }
            }
        }

        stage('Quality Gate Check') {
            steps {
                timeout(time: 1, unit: 'HOURS') {
                    waitForQualityGate abortPipeline: false, credentialsId: 'sonar-token'
                }
            }
        }

        stage('Build') {
            steps {
                sh "mvn package"
            }
        }

        stage('Publish To Nexus') {
            steps {
                withMaven(globalMavenSettingsConfig: 'devopsshack', maven: 'maven3', traceability: true) {
                    sh "mvn deploy"
                }
            }
        }

        stage('Docker Image Build & Tag') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker-cred') {
                        sh "docker build -t abhishekfpt/bankapp:$IMAGE_TAG ."
                    }
                }
            }
        }

        stage('Scan Image') {
            steps {
                sh "trivy image --format table -o image-report.html abhishekfpt/bankapp:$IMAGE_TAG"
            }
        }

        stage('Push Docker Image') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker-cred') {
                        sh "docker push abhishekfpt/bankapp:$IMAGE_TAG"
                    }
                }
            }
        }

        stage('Update Manifest File in Mega-Project-CD') {
            steps {
                script {
                    // Clean workspace before starting
                    cleanWs()

                    withCredentials([usernamePassword(credentialsId: 'git', usernameVariable: 'GIT_USERNAME', passwordVariable: 'GIT_PASSWORD')]) {
                        sh '''
                            # Clone the Mega-Project-CD repository
                            git clone https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/abhi21993/mega-project-cd.git

                            # Update the image tag in the manifest.yaml file
                            cd Mega-Project-CD
                            sed -i "s|abhishekfpt/bankapp:.*|abhishekfpt/bankapp:${IMAGE_TAG}|" Manifest/manifest.yaml

                            # Confirm changes
                            echo "Updated manifest file contents:"
                            cat Manifest/manifest.yaml

                            # Commit and push the changes
                            git config user.name "Jenkins"
                            git config user.email "jenkins@example.com"
                            git add Manifest/manifest.yaml
                            git commit -m "Update image tag to ${IMAGE_TAG}"
                            git push origin main
                        '''
                    }
                }
            }
        }
    }
}

Check the nexus repository

Every time the pipeline will get executed the image version will get change

Note: Here in jenkins pipeline add the credential of git for cd repo

Add the username and password with seperated parameters

Use the clone of cd repository and change some file

This will update the version of docker image in cd repository

Phase 6: Jenkins Pipeline Deployment to kubernetes(CD)

Before deployment make sure to create RBAC.

For Jenkins to perform deployments, manage storage, and configure persistent volumes, it needs authorization for those actions.
If you don’t set up RBAC:
- Jenkins will be denied when trying to create, update, or delete resources.
- Kubernetes will return “Forbidden” errors.
- By configuring:
  - A Role (namespace-scoped) allows Jenkins to manage resources (deployments, secrets, configs, autoscaling, etc.) in a specific namespace.
  - A ClusterRole (cluster-scoped) gives Jenkins extra permissions it may need for global resources, e.g. storage classes and persistent volumes shared across many namespaces.

Create Service account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: webapps

Why service account needed?

A ServiceAccount is a special identity for applications (like Jenkins) running in a pod, allowing them to securely interact with the Kubernetes API.
When Jenkins interacts with Kubernetes (to deploy apps, create resources, etc.), it does so through the Kubernetes API
Kubernetes requires every API call to be authenticated and authorized.

How we can utilize jenkins service account for authentication for doing deployment?

Every ServiceAccount has a token—a secure string (JWT) Jenkins uses to prove its identity to Kubernetes.
Jenkins (by itself or via plugins like Kubernetes CLI, KubeDeploy, etc.) uses this token to authenticate API calls.
This token is safer: It only grants access to what Jenkins needs, rather than giving Jenkins admin rights to the whole cluster.

Generate the token from the link

https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#:~:text=To%20create%20a%20non%2Dexpiring,with%20that%20generated%20token%20data.

Add the token in jenkins pipeline

Create Role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins-role
  namespace: webapps
rules:
  # Permissions for core API resources
  - apiGroups: [""]
    resources:
      - secrets
      - configmaps
      - persistentvolumeclaims
      - services
      - pods
    verbs: ["get", "list", "watch", "create", "update", "delete","patch"]

  # Permissions for apps API group
  - apiGroups: ["apps"]
    resources:
      - deployments
      - replicasets
      - statefulsets
    verbs: ["get", "list", "watch", "create", "update", "delete","patch"]

  # Permissions for networking API group
  - apiGroups: ["networking.k8s.io"]
    resources:
      - ingresses
    verbs: ["get", "list", "watch", "create", "update", "delete","patch"]

  # Permissions for autoscaling API group
  - apiGroups: ["autoscaling"]
    resources:
      - horizontalpodautoscalers
    verbs: ["get", "list", "watch", "create", "update", "delete","patch"]

Create Role Binding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-rolebinding
  namespace: webapps
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins-role
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: webapps

Create Cluster role

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-cluster-role
rules:
  # Permissions for persistentvolumes
  - apiGroups: [""]
    resources:
      - persistentvolumes
    verbs: ["get", "list", "watch", "create", "update", "delete"]
  # Permissions for storageclasses
  - apiGroups: ["storage.k8s.io"]
    resources:
      - storageclasses
    verbs: ["get", "list", "watch", "create", "update", "delete"]
  # Permissions for ClusterIssuer
  - apiGroups: ["cert-manager.io"]
    resources:
      - clusterissuers
    verbs: ["get", "list", "watch", "create", "update", "delete"]

Create Cluster role binding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins-cluster-role
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: webapps

In master node install

nginx controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

install cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml

We have put the service as clusterIP, so that it should expose to world with the help of ingress

Write jenkins file for deployment

note: change the cluster-name and severUrl of your EKS

pipeline {
    agent any

    stages {
        stage('Git Checkout') {
            steps {
                git branch: 'main', credentialsId: 'git', url: 'https://github.com/abhi21993/mega-project-cd.git'
            }
        }

        stage('Kubernetes Deployment') {
            steps {
                withKubeConfig(caCertificate: '', clusterName: 'abhiproject-cluster', contextName: '', credentialsId: 'k8-token', namespace: 'webapps', restrictKubeConfigAccess: false, serverUrl: 'https://952FB702C508F688D873376083B31DF5.gr7.ap-south-1.eks.amazonaws.com') {
                    sh "kubectl apply -f Manifest/manifest.yaml -n webapps"
                    sh "kubectl apply -f Manifest/HPA.yaml "
                    sleep 30
                    sh "kubectl get pods -n webapps"
                    sh "kubectl get service -n webapps"
                }
            }
        }
    }
}

When the pipeline will run successfully
check with command kubectl get ingress -n webapps

You will get the Ingress LoadBalancer URL in the address field. Simply map this URL to your domain name, and you will be able to see the application output.

DevOps Project: CI/CD Pipeline Setup on AWS EKS