Project Deployment Flow:

Tech stack used in this project:

• GitHub (Code)

• Docker (Containerization)

• Jenkins (CI)

• OWASP (Dependency check)

• SonarQube (Quality)

• Trivy (Filesystem Scan)

• ArgoCD (CD)

• AWS EKS (Kubernetes)

• Helm (Monitoring using grafana and prometheus)

Mistakes I Made & What I Learned

Mistake: Incorrect ALB Configuration for EKS

• Description: I faced difficulties with the Application Load Balancer (ALB) setup, which led to issues with routing and service discovery in AWS EKS.

• Solution: I revisited the ALB configuration, following best practices for routing and integrating it with Kubernetes services to ensure proper traffic management.

Mistake: Ignoring IAM Permissions in AWS EKS

• Description: I neglected to properly configure IAM roles and security policies, leading to access and permission issues when deploying on EKS.

• Solution: After researching the required IAM configurations, I ensured that all roles and security groups were set correctly, granting the necessary permissions to the services.

Mistake: Not Properly Managing Secrets

• Description: I initially mishandled sensitive credentials, which led to security risks and potential exposure of private data.

• Solution: I adopted best practices for managing secrets using AWS Secrets Manager and Kubernetes Secrets to securely store and access sensitive information.

Mistake: Skipping Documentation of the Deployment Process

• Description: I didn’t document the deployment process properly, making it hard to revisit and explain how the app was set up later.

• Solution: I documented the entire deployment process step-by-step, including specific configurations and setup instructions, for future reference and for anyone working with the project.

Mistake: Missing Detailed Logging and Error Handling

• Description: My application initially lacked proper logging and error handling, making it difficult to diagnose issues during runtime.

• Solution: I added comprehensive logging and error handling across all layers of the application, which allowed me to easily trace and resolve issues as they arose.

---

Prerequisites to implement this project:

Note:

This project will be implemented in the North California region (us-west-1).

• Create 1 master machine on AWS with 2 CPUs, 8GB of RAM (t2.large), and 29 GB of storage, and install Docker on it.

• Open the below ports in security group of master machine and also attach same security group to Jenkins worker node (We will create worker node shortly)

  

Note

We are creating this master machine because we will configure Jenkins master, eksctl, and EKS cluster creation from here.

Install & configure Docker by using the below command; "NewGrp docker" will refresh the group config, hence there is no need to restart the EC2 machine.

sudo apt-get update

sudo apt-get install docker.io -y
sudo usermod -aG docker ubuntu && newgrp docker

• Install and configure Jenkins

sudo apt update -y
sudo apt install fontconfig openjdk-17-jre -y

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key

echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null

sudo apt-get update -y
sudo apt-get install jenkins -y

• Now, access Jenkins Master on the browser on port 8080 and configure it.

• Create EKS Cluster on AWS (Master machine)

  • IAM user with access keys and secret access keys

  • AWSCLI should be configured (Setup AWSCLI)

    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    sudo apt install unzip
    unzip awscliv2.zip
    sudo ./aws/install
    aws configure

• Install kubectl (Setup kubectl )

    curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
    chmod +x ./kubectl
    sudo mv ./kubectl /usr/local/bin
    kubectl version --short --client

• Install eksctl (Setup eksctl)

    curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
    sudo mv /tmp/eksctl /usr/local/bin
    eksctl version

• Create EKS Cluster

    eksctl create cluster --name=wearsphere \
                        --region=ap-south-1 \
                        --version=1.30 \
                        --without-nodegroup

• Associate IAM OIDC Provider

    eksctl utils associate-iam-oidc-provider \
      --region ap-south-1 \
      --cluster wearsphere\
      --approve

• Create Node

    eksctl create nodegroup --cluster=wearsphere\
                         --region=ap-south-1 \
                         --name=wearsphere\
                         --node-type=t2.large \
                         --nodes=2 \
                         --nodes-min=2 \
                         --nodes-max=2 \
                         --node-volume-size=29 \
                         --ssh-access \
                         --ssh-public-key=eks-nodegroup-key

Creating Dockerfile for frontend

FROM node:18 AS builder

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

#Stage 2
FROM node:18-alpine

WORKDIR /app

COPY --from=builder /app .

EXPOSE 5173

CMD ["npm","run","dev"]

docker build -t wearsphere-frontend .

docker run -d -p 5173:5173 wearsphere-frontend:latest

Docker file for Backend

Change this with your EC2 private ip

docker build -t wearsphere-backend .

docker run -d -p 4000:4000 wearsphere-backend:latest

Backend Running Fine too

Creating Dockerfile for Admin Panel

Change localhost with your public ip address of EC2

Admin Panel is running too on port 5174

---

Applying docker-compose.yml

inside .env of frontend, add your public ip of ec2 instead of mine

docker-compose up --build -d

For Adding Sample Products

Create seed.js inside backend folder

import mongoose from "mongoose";
import dotenv from "dotenv";
import productModel from "./models/productModel.js";

dotenv.config();

const connectDB = async () => {
  await mongoose.connect(process.env.MONGODB_URI);
  console.log("MongoDB connected!");
};

const seedProducts = async () => {
  try {
    await connectDB();

    const products = [
      {
        name: "Cool Hoodie",
        description: "Comfortable and stylish",
        price: 1299,
        image: ["https://dummyimage.com/300"],
        category: "clothing",
        subCategory: "hoodie",
        sizes: ["S", "M", "L"],
        bestseller: true,
        date: Date.now()
      },
      {
        name: "Graphic T-Shirt",
        description: "Trendy streetwear tee",
        price: 699,
        image: ["https://dummyimage.com/301"],
        category: "clothing",
        subCategory: "tshirt",
        sizes: ["M", "L"],
        bestseller: false,
        date: Date.now()
      }
    ];

    await productModel.deleteMany(); // Optional: Clears existing
    await productModel.insertMany(products);
    console.log("Sample products seeded! 🎉");
    process.exit();
  } catch (err) {
    console.error(err);
    process.exit(1);
  }
};

seedProducts();

Then go inside the wearsphere-backend container and run seed. js

docker exec -it wearsphere-backend sh

ls

node seed.js

And Sample products Starts Coming

---

Push Images to Docker Hub

docker login

docker tag wearsphere-frontend dakshsawhneyy/wearsphere-frontend
docker push dakshsawhneyy/wearsphere-frontend

docker tag wearsphere-backend dakshsawhneyy/wearsphere-backend
docker push dakshsawhneyy/wearsphere-backend

docker tag wearsphere-admin dakshsawhneyy/wearsphere-admin
docker push dakshsawhneyy/wearsphere-admin

---

Setting up Jenkins

Go to Jenkins Dashboard

Add Credentials of Docker, Gmail, SonarQube

Go to SonarQube

Copy this token into Jenkins credentials

Now go to your gmail account > manage account

Create one app password and paste this in jenkins

Now, installing plugins and setting up system environments

Go to manage jenkins > Plugin and install

• OWASP Dependency

• Stage view

• SonarQube Scanner

• SonarQube Quality Gates

Go to Manage Jenkins > tools

Now, go to manage jenkins > system

Go to SonarQube and add webhook

• Install and configure SonarQube

docker run -itd --name SonarQube-Server -p 9000:9000 sonarqube:lts-community

• Install Trivy

sudo apt-get install wget apt-transport-https gnupg lsb-release -y
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update -y
sudo apt-get install trivy -y

Creating a Pipeline in Jenkins

writing groovy pipeline script - Only for CI (initially)

Library('@Shared') _
pipeline {
    agent any
    environment{
        SONAR_HOME = tool 'Sonar'
    }

    stages{
        stage("WorkSpace Empty"){
            steps{
                script{
                    cleanWs()
                }
            }
        }
        stage('Git Code Clone'){
            steps{
                script{
                    clone("https://github.com/dakshsawhneyy/WearSphere-Ecommerce-MERN.git","master")
                }
            }
        }
        stage("Trivy: File Scan"){
            steps{
                script{
                    trivy_scan()
                }
            }
        }
        stage("OWASP: Dependency check"){
            steps{
                script{
                    owasp_dependency()
                }
            }
        }
        stage("SonarQube: Code Analysis"){
            steps{
                script{
                    sonarqube_analysis("Sonar","wearsphere","wearsphere")
                }
            }
        }
        stage("SonarQube: Code Quality Gates"){
            steps{
                script{
                    sonarqube_code_quality()
                }
            }
        }
        stage("Docker: Build Images"){
            steps{
                script{
                    dir('backend'){
                        docker_build("dakshsawhneyy","wearsphere-backend","latest")
                    }
                    dir('frontend'){
                        docker_build("dakshsawhneyy","wearsphere-frontend","latest")
                    }
                    dir('admin'){
                        docker_build("dakshsawhneyy","wearsphere-admin","latest")
                    }
                }
            }
        }
        stage("Docker: Push to DockerHub"){
            steps{
                script{
                    docker_push("wearsphere-backend","latest","dakshsawhneyy") 
                    docker_push("wearsphere-frontend","latest","dakshsawhneyy")
                    docker_push("wearsphere-admin","latest","dakshsawhneyy")
                }
            }
        }
    }
}

---

For making it publicly accessible, we need INGRESS CONTROLLER

Our cluster is isolated, basically, so to make it accessible to outside world, we need Ingress Controller

So for installing Ingress Controller, we need to install HELM !!

Install AWS Load Balancer

COPY

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json
aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam_policy.json
eksctl utils associate-iam-oidc-provider --region=ap-south-1 --cluster=three-tier-cluster --approve
eksctl create iamserviceaccount --cluster=three-tier-cluster --namespace=kube-system --name=aws-load-balancer-controller --role-name AmazonEKSLoadBalancerControllerRole --attach-policy-arn=arn:aws:iam::626072240565:policy/AWSLoadBalancerControllerIAMPolicy --approve --region=ap-south-1

Deploy AWS Load Balancer Controller

COPY

sudo snap install helm --classic
helm repo add eks https://aws.github.io/eks-charts
helm repo update eks
helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=three-tier-cluster --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
kubectl get deployment -n kube-system aws-load-balancer-controller

For deploying this on your custom domain, go to your domain purchase platform (in my case, it’s NameCheap.com)

go to browser and try to open http://wearsphere.cctlds.online

Backend pod is crashing because it is unable to connect to backend

go inside backend folder ./backend/.env

Our site is deployed on our custom domain 🎉🎉🎉

---

• Install and Configure ArgoCD

  • Create argocd namespace

    kubectl create namespace argocd

• Apply Argo manifest

    kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

• Make sure all pods are running in argocd namespace

    watch kubectl get pods -n argocd

• Install argocd CLI

    sudo curl --silent --location -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/v2.4.7/argocd-linux-amd64

• Provide executable permission

    sudo chmod +x /usr/local/bin/argocd

• Check argocd services

    kubectl get svc -n argocd

• Change argocd server's service from ClusterIP to NodePort

    kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "NodePort"}}'

• Confirm service is patched or not

    kubectl get svc -n argocd

• Go to Browser → <your-ip-of-node>/port

  

  • Fetch the initial password of argocd server

COPY

        kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

• Username: admin

• Now, go to User Info and update your argocd password

  

  

  

Adding our own eks cluster to argocd for application deployment using cli

• Login to Argo from CLI

• COPY

         argocd login 52.53.156.187:32738 --username admin
  

  [!Tip] 52.53.156.187:32738 --> This should be your argocd url

  

  • Check how many clusters are available in argocd

COPY

    argocd cluster list

• Get your cluster name

COPY

    kubectl config get-contexts

• 

• Add your cluster to argocd

COPY

    argocd cluster add  dakshsawhneyy@wearsphere.ap-south-1.eksctl.io --name wearsphere

[!Tip] dakshsawhneyy@wearsphere.ap-south-1.eksctl.io --> This should be your EKS Cluster Name.

• Once your cluster is added to argocd, go to argocd console Settings --> Clusters and verify it

• 

• Now, go to Applications and click on New App

[!Important] Make sure to click on the Auto-Create Namespace option while creating argocd application

• 

• Congratulations, your application is deployed on AWS EKS Cluster

  

Argo CD is working perfectly fine 🎉🎉🎉

---

Updating Jenkins file with CD

@Library('Shared') _
pipeline {
    agent any
    environment{
        SONAR_HOME = tool 'Sonar'
    }

    stages{
        stage("WorkSpace Empty"){
            steps{
                script{
                    cleanWs()
                }
            }
        }
        stage('Git Code Clone'){
            steps{
                script{
                    clone("https://github.com/dakshsawhneyy/WearSphere-Ecommerce-MERN.git","master")
                }
            }
        }
        stage("Trivy: File Scan"){
            steps{
                script{
                    trivy_scan()
                }
            }
        }
        stage("OWASP: Dependency check"){
            steps{
                script{
                    owasp_dependency()
                }
            }
        }
        stage("SonarQube: Code Analysis"){
            steps{
                script{
                    sonarqube_analysis("Sonar","wearsphere","wearsphere")
                }
            }
        }
        stage("SonarQube: Code Quality Gates"){
            steps{
                script{
                    sonarqube_code_quality()
                }
            }
        }
        stage("Docker: Build Images"){
            steps{
                script{
                    dir('backend'){
                        docker_build("dakshsawhneyy","wearsphere-backend","latest")
                    }
                    dir('frontend'){
                        docker_build("dakshsawhneyy","wearsphere-frontend","latest")
                    }
                    dir('admin'){
                        docker_build("dakshsawhneyy","wearsphere-admin","latest")
                    }
                }
            }
        }
        stage("Docker: Push to DockerHub"){
            steps{
                script{
                    docker_push("wearsphere-backend","latest","dakshsawhneyy") 
                    docker_push("wearsphere-frontend","latest","dakshsawhneyy")
                    docker_push("wearsphere-admin","latest","dakshsawhneyy")
                }
            }
        }
        stage("Update Kubernetes Manifests") {
            steps {
                script {
                    k8s_manifests('latest') 
                }
            }
        }
    }
    post {
        success {
            emailext (
                to: 'dakshsawhney2@example.com',  
                subject: "SUCCESS: WearSphere Pipeline - ${currentBuild.fullDisplayName}",
                body: """
                    The Jenkins pipeline for the WearSphere project has successfully completed.
                    Build URL: ${currentBuild.absoluteUrl}
                    Build Status: SUCCESS
                """
            )
        }
        failure {
            emailext (
                to: 'dakshsawhney2@example.com',  // Replace with the recipient's email
                subject: "FAILURE: WearSphere Pipeline - ${currentBuild.fullDisplayName}",
                body: """
                    The Jenkins pipeline for the WearSphere project has failed.
                    Build URL: ${currentBuild.absoluteUrl}
                    Build Status: FAILURE
                """
            )
        }
    }
}

---

How to monitor EKS cluster, kubernetes components and workloads using prometheus and grafana via HELM (On Master machine)

• Install Helm Chart

COPY

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3

COPY

chmod 700 get_helm.sh

COPY

./get_helm.sh

• Add Helm Stable Charts for Your Local Client

COPY

helm repo add stable https://charts.helm.sh/stable

• Add Prometheus Helm Repository

COPY

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

• Create Prometheus Namespace

COPY

kubectl create namespace prometheus

COPY

kubectl get ns

• Install Prometheus using Helm

COPY

helm install stable prometheus-community/kube-prometheus-stack -n prometheus

• Verify prometheus installation

COPY

kubectl get pods -n prometheus

• Check the services file (svc) of the Prometheus

COPY

kubectl get svc -n prometheus

• Expose Prometheus and Grafana to the external world through Node Port

[!Important] change it from Cluster IP to NodePort after changing make sure you save the file and open the assigned nodeport to the service.

COPY

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

• Verify service

COPY

kubectl get svc -n prometheus

• Now, let’s change the SVC file of the Grafana and expose it to the outer world

COPY

kubectl edit svc stable-grafana -n prometheus

• Check grafana service

COPY

kubectl get svc -n prometheus

• Get a password for grafana

COPY

kubectl get secret --namespace prometheus stable-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

[!Note] Username: admin

• Now, view the Dashboard in Grafana

  

  

  

Clean Up

• Delete eks cluster

COPY

eksctl delete cluster --name=wearsphere --region=ap-south-1