Automating EKS Deployment with Terraform and GitHub Actions – The DevOps Way

https://github.com/user-attachments/assets/b93fa528-6778-4c97-94ee-6e36c7f4fa8e

Project Overview

This project outlines the step-by-step process of setting up a CI/CD pipeline using GitHub Actions. It shows how to automate an application's build, test, and deployment to Kubernetes using tools like Docker, Trivy, SonarQube, and Terraform. The project also highlights the integration of AWS roles for managing cloud resources and Kubernetes clusters.

---

Prerequisites

Before diving into this project, here are some skills and tools you should be familiar with:

• Terraform is installed on your machine.

• A GitHub account.

• A GitHub personal access token with the necessary permissions to create repositories.

⚠️ Important:

1. Make sure First you will create a .pem key manually from the AWS console. i.e "MYLABKEY.pem" because it will be used for creating EC2 VMs and EKS cluster.

2. Copy MYLABKEY.pem in the terraform directory (01.Code_IAC_Selfhosted-Runner-and-Trivy and 03.Code_IAC_Terraform_box ) as below your terraform code

3. Generate the Github Token

ls 
\Learning_GitHub_Action\01.Github_Action_DevOps-Project\Terraform_Code_Infra_setup


Mode                 LastWriteTime         Length Name                                                                                                                                                                                              
----                 -------------         ------ ----                                                                                                                                                                                              
dar--l          17/04/25  12:48 PM                .terraform                                                                                                                                                                                        
dar--l          21/04/25  12:34 PM                00.Code_IAC-github-repo                                                                                                                                                                           
dar--l          21/04/25  12:34 PM                01.Code_IAC_Selfhosted-Runner-and-Trivy                                                                                                                                                           
dar--l          21/04/25   1:38 PM                02.Code_IAC_SonarQube                                                                                                                                                                             
dar--l          21/04/25  12:34 PM                03.Code_IAC_Terraform_box                                                                                                                                                                         
-a---l          20/08/24   1:45 PM            493 .gitignore                                                                                                                                                                                                                                                                                                                                    
-a---l          21/04/25   1:59 PM          18225 AboutThis Project.md                                                                                                                                                                              
-a---l          19/04/25   8:48 PM           1309 main.tf

• Clone the repository for the Terraform code

  💡 Note: Replace GitHub Token, resource names and variables as per your requirement in terraform code

  • For github Repo Token value to be updated in file - 00.Code_IAC-github-repo/variables.tf (i.e default- xxxxxx*)

  • For EC2 VM - 01.Code_IAC_Selfhosted-Runner-and-Trivy/main.tf (i.e keyname- MYLABKEY) - 03.Code_IAC_Terraform_box/main.tf (i.e keyname- MYLABKEY)

  • For Cluster name - 03.Code_IAC_Terraform_box/k8s_setup_file/main.tf (i.e balraj*).

  • For Node Pod - 03.Code_IAC_Terraform_box/k8s_setup_file/variable.tf (i.e MYLABKEY*)

• Set up your GitHub token:

  • Create a new GitHub personal access token with the repo scope at https://github.com/settings/tokens.

  • Then set it as an environment variable (DO NOT commit your token to version control):

    # For Linux/macOS
    export GITHUB_TOKEN=your_github_token

    # For Windows Command Prompt
    set GITHUB_TOKEN=your_github_token

    # For Windows PowerShell (I used this one)
    # $env:GITHUB_TOKEN="your_github_token"
    $env:TF_VAR_github_token = "your-github-personal-access-token"

• Test and verify with curl again in powershell terminal:

    $headers = @{
     Authorization = "token $env:TF_VAR_github_token"
    }
    Invoke-WebRequest -Uri "https://api.github.com/user" -Headers $headers
  

  • You should see your GitHub user info in JSON, not "Bad credentials".

---

Key Points

1. GitHub Actions Overview:

   • GitHub Actions is used as the CI/CD tool for this project.

   • It eliminates the need for setting up and maintaining Jenkins servers by providing managed runners.

2. Pipeline Stages:

   • Compile: Builds the application.

   • Security Checks: Scans for vulnerabilities using Trivy and GitLeaks.

   • Unit Testing: Executes test cases to ensure code quality.

   • Build and Publish Docker Image: Builds a Docker image and uploads it as an artifact.

   • Deploy to Kubernetes: Deploys the application to an EKS cluster using Terraform.

3. Tools and Technologies Used:

   • GitHub Actions: CI/CD automation.

   • Docker: Containerization of the application.

   • Trivy: Security scanning for vulnerabilities.

   • GitLeaks: Detects hardcoded secrets in the source code.

   • SonarQube: Code quality analysis.

   • AWS CLI: Manages AWS resources.

   • Terraform: Infrastructure as Code (IaC) for provisioning EKS clusters.

   • Kubernetes: Orchestrates containerized applications.

4. Why Use This Project:

   • Automates the software delivery process.

   • Ensures code quality and security through automated checks.

   • Simplifies deployment to Kubernetes clusters.

   • Demonstrates best practices for CI/CD pipelines.

5. Takeaways:

   • Understanding of GitHub Actions and its capabilities.

   • Hands-on experience with integrating security tools like Trivy and GitLeaks.

   • Knowledge of deploying applications to Kubernetes using Terraform.

   • Insights into managing AWS resources with AWS CLI.

---

Step-by-Step Process

Setting Up the Infrastructure

I have created a Terraform code to set up the entire infrastructure, including the installation of required applications, tools, and the EKS cluster automatically created.

• ⇒ Docker Install

• ⇒ SonarQube Install

• ⇒ Trivy Install

• ⇒ Terraform Install

• ⇒ EKS Cluster Setup

💡 Note: ⇒ EKS cluster creation will take approx. 10 to 15 minutes.

To Create EC2 Instances

First, we'll create the necessary virtual machines using terraform code.

Below is a Terraform code:

Once you clone repo, then go tothe folder "01.Github_Action_DevOps-Project/Terraform_Code_Infra_setup" and run the terraform command.

cd 01.Github_Action_DevOps-Project/Terraform_Code_Infra_setup

$ ls

 00.Code_IAC-github-repo/   01.Code_IAC_Selfhosted-Runner-and-Trivy/   02.Code_IAC_SonarQube/   03.Code_IAC_Terraform_box/  'AboutThis Project.md'   main.tf

💡 Note: ⇒ Make sure to run main.tf which is located outside of the folder. I have created the code in such a way that a single file will call all of the folders.

 ls -la
total 72
-rw-r--r-- 1 bsingh 1049089   493 Aug 20  2024  .gitignore
drwxr-xr-x 1 bsingh 1049089     0 Apr 21 12:34  00.Code_IAC-github-repo/
drwxr-xr-x 1 bsingh 1049089     0 Apr 21 12:34  01.Code_IAC_Selfhosted-Runner-and-Trivy/
drwxr-xr-x 1 bsingh 1049089     0 Apr 21 13:38  02.Code_IAC_SonarQube/
drwxr-xr-x 1 bsingh 1049089     0 Apr 21 12:34  03.Code_IAC_Terraform_box/
-rw-r--r-- 1 bsingh 1049089 21284 Apr 21 14:44 'AboutThis Project.md'
-rw-r--r-- 1 bsingh 1049089  1309 Apr 19 20:48  main.tf

You need to run main.tf file using following terraform command.

Now, run the following command.

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply 
# Optional <terraform apply --auto-approve>

---

Once you run the Terraform command, we will verify the following things to ensure everything is set up properly using Terraform.

Inspect the Cloud-Init logs:

Once connected to the EC2 instance then you can check the status of the user_data Script by inspecting the log files.

# Primary log file for cloud-init
sudo tail -f /var/log/cloud-init-output.log
                    or 
sudo cat /var/log/cloud-init-output.log | more

🔍- If the user_data script runs successfully, you will see output logs and any errors encountered during execution.

🔍- If there’s an error, this log will provide clues about what failed.

• Verify the Outcome of "cloud-init-output.log"

Verify the Installation

• [x] Docker version

ubuntu@ip-172-31-95-197:~$ docker --version
Docker version 24.0.7, build 24.0.7-0ubuntu4.1


docker ps -a
ubuntu@ip-172-31-94-25:~$ docker ps

• [x] Trivy version

ubuntu@ip-172-31-89-97:~$ trivy version
Version: 0.55.2

• [x] Terraform version

ubuntu@ip-172-31-89-97:~$ terraform version
Terraform v1.9.6
on linux_amd64

• [x] eksctl version

ubuntu@ip-172-31-89-97:~$ eksctl version
0.191.0

• [x] kubectl version

ubuntu@ip-172-31-89-97:~$ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2

• [x] AWS CLI version

ubuntu@ip-172-31-89-97:~$ aws version
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
  aws help
  aws <command> help
  aws <command> <subcommand> help

Verify the EKS Cluster installation

• Will take a putty session from Terraform EC2

• On the terraform virtual machine, go to the directory k8s_setup_file and open the file cat apply.log to verify whether the cluster is created or not.

• Will verify the cluster status from

  • sudo cat /var/log/cloud-init-output.log | more or

  • cat /home/ubuntu/k8s_setup_file/apply.log

    ubuntu@ip-172-31-90-126:~/k8s_setup_file$ pwd
    /home/ubuntu/k8s_setup_file
    ubuntu@ip-172-31-90-126:~/k8s_setup_file$ cd ..

• 

• After Terraform deploys on the instance, it's time to set up the cluster. If you log out of the SSH session, reconnect and run the following command:

    aws eks update-kubeconfig --name <cluster-name> --region 
    <region>
  

• Once the EKS cluster is set up, you need to run the following command to interact with EKS.

    aws eks update-kubeconfig --name balraj-cluster --region us-east-1
  

⚠️ Important:
The aws eks update-kubeconfig command is used to configure your local kubectl tool to interact with an Amazon EKS (Elastic Kubernetes Service) cluster. It updates or creates a kubeconfig file that contains the necessary authentication information to allow kubectl to communicate with your specified EKS cluster.

What happens when you run this command:
The AWS CLI retrieves the required connection information for the EKS cluster (such as the API server endpoint and certificate) and updates the kubeconfig file located at ~/.kube/config (by default). It configures the authentication details needed to connect kubectl to your EKS cluster using IAM roles. After running this command, you will be able to interact with your EKS cluster using kubectl commands, such as kubectl get nodes or kubectl get pods.

kubectl get nodes
kubectl cluster-info
kubectl config get-contexts

---

Verify GitHub Repo and GitHub Actions

• Verify that the GitHub repository is created and initialise it since we are using Terraform.

• 

• Verify a .github/workflows directory was created along with two YAML files for the pipeline.

• 

Adding a Virtual Machine as a Runner

• I'll be using a self-hosted runner to execute the pipeline.

• Configure the runner by following the commands provided in GitHub's settings.

       Go to "GithubAction_DevOps_Projects"
       Click on settings
       then select the actions and choose "runners"
  

• 

• Click on new self-hosted runner and select Linux

• Notedown the token value somewhere as we need to in runner VM.

• 

  Take a putty session of runner EC2

• Go to actions-runner the folder

• 

• Update/Paste the token value here as mentioned in the screenshot.

• 

• Change the execution mode for the script and run it.

• chmod +x selfhost-runner.sh

💡 Note:

Take note of the token value from here and paste it into the script in runner at the following spot. This ensures that the script executes successfully with the necessary permissions. Once you've finished, save your modifications and run the script to test whether it works as planned.

Troubleshooting:

• I am getting below error message while executing the file.

• 

Fix/Solution:

• I try explicitly invoking the bash interpreter:

    bash ./selfhost-runner.sh
  

• The solution is to remove these carriage return characters using the dos2unix command:

  1. Install dos2unix if you haven't already:

    sudo apt-get update
    sudo apt-get install dos2unix

2. Run dos2unix on selfhost-runner.sh script:

    dos2unix selfhost-runner.sh

3. Try running the script again:

    ./selfhost-runner.sh

💡 Idea: This should now execute correctly because the problematic carriage return characters will have been removed

It works :-) And I can execute the file.

Setup SonarQube

• Go to SonarQube EC2 and notedown the Public IPAddress and open a new browser.

• Access SonarQube via http://<your-server-ip>:9000.

• 

• 

  💡 Note: When you access the URL above, you will be prompted to log in. Use "admin/admin" for the first-time login, and you will be asked to change the password. Once you change it, make sure to create a strong and memorable password. After that, you will have full access to the system's features and settings._

Create a token in SonarQube

• Go to Administration\>Security\>Users\>Create a new token

Configure Secrets and Variables in GitHub Repo.

Go to Repo `GithubAction_DevOps_Projects`
Click on `settings`
Click on `Secrets and Variables`
Select `Actions`.

💡 Note:

You have to update all the required tokens and secrets value here. Part of Terraform code, I have already created a dummy values, which needs to be replaced. Once you have replaced the dummy values with the actual tokens and secrets, ensure that you test the configuration thoroughly to confirm that everything functions as expected. This will help prevent any issues during deployment and maintain the integrity of your infrastructure.

• To Update Sonar URL

• 

• To update the EKS_KUBECONFIG secret

  • Take a putty session of the Terraform EC2 instance

  • Run the command cat ~/.kube/config

  • Copy the whole content and paste it into the secret.

Attach Role to Runner EC2

• Select the EC2 VM and click on the actions > security\> Mofify IAM Roles on the runner.

• 

• Select the role Role_k8_Cluster_profile

• 

• Click on update IAM Role.

Writing the CI/CD Pipeline

• Compile Stage:

  • Use actions/checkout to clone the repository.

  • Set up the required environment (e.g., JDK 17 for Java projects).

  • Compile the application using build tools like Maven.

• Security Checks:

  • Install and run Trivy to scan for vulnerabilities in Docker images.

  • Use GitLeaks to detect hardcoded secrets in the source code.

• Unit Testing:

  • Execute test cases to validate the application.

• Build and Publish Docker Image:

  • Build a Docker image using docker build.

  • Push the image to a container registry or upload it as an artifact.

• Deploy to Kubernetes:

  • Use Terraform to provision an EKS cluster.

  • Deploy the application using Kubernetes manifests.

• Here are the complete CICD Pipeline details

Verify the Docker Image

Verify code coverage in SonarQube

Verify pipeline Status

Verify the pods in the runner VM

Verify Application Status

Environment Cleanup:

• The following resources were created as part of this project.

• 

To delete a deployment:

• I've created a Github Action to destroy the Kubernetes deployment and services.

• 

• Delete all deployment/Service:

  • • In GitHub actions, click on the second pipeline to delete the deployment and service.

      • 

        

      • Here is the complete CICD- Pipeline to destroy Deployment and Services

To delete AWS EKS cluster

• Log in to the Terraform EC2 instance and change the directory to /k8s_setup_file, and run the following command to delete the cluster.

  •   sudo su - ubuntu
      cd /k8s_setup_file
      sudo terraform destroy --auto-approve
    

Troubleshooting:

• I am getting below error message while running the Terraform destroy.

• 

Fix/Solution:

• I noticed that permission is set to root for terraform dirctory. we have to take ownership first and then try to delete it.

• 

• Run the following command to take ownership

    sudo chown -R ubuntu:ubuntu /home/ubuntu/k8s_setup_file/.terraform*
  

• I was still getting an error message while running the destroy command.

• 

• I ran the following command again for the entire Terraform folder.

    sudo chown -R ubuntu:ubuntu /home/ubuntu/k8s_setup_file/terraform*
  

• Rerun the destroy command, and this time it works :-)

• 

To delete the Virtual machine.

Go to folder "01.Github_Action_DevOps-Project/Terraform_Code_Infra_setup" and run the terraform command.

• 00.Code_IAC-github-repo

• 01.Code_IAC_Selfhosted-Runner-and-Trivy - 02.Code_IAC_SonarQube

• 03.Code_IAC_Terraform_box

    Terraform destroy --auto-approve
  

💡 Note:

You must use this command from each folder in order to destroy the entire infrastructure.

---

Why Use This Project

• Automation: Reduces manual effort in building, testing, and deploying applications.

• Security: Ensures code and container security through automated scans.

• Scalability: Deploys applications to scalable Kubernetes clusters.

• Best Practices: Demonstrates industry-standard CI/CD practices.

---

Conclusion

This project offers a complete guide to setting up a CI/CD pipeline with GitHub Actions. By using tools like Docker, Trivy, SonarQube, and Terraform, it ensures a secure and efficient software delivery process. The use of AWS CLI and Kubernetes shows how to deploy applications to cloud-native environments. This project is a helpful resource for DevOps engineers aiming to implement modern CI/CD pipelines.

Ref Link:

• Youtube VideoLink

• Clearfile-content cache in visualstudio code

• managing-GitHub-access-tokens

• GitHub Actions Markget Place

• download-a-build-artifact

• upload-a-build-artifact