To create and manage CI/CD pipelines with Jenkins on an AWS EC2 instance using a master-slave architecture, you'll need to follow these steps. This guide covers setting up Jenkins, configuring a master-slave setup, and updating security groups to allow communication.

Prerequisites

AWS EC2 instance (Ubuntu-based)
Jenkins installed on EC2 master node
Slave EC2 instances set up with the required environment (Java, Jenkins agent)
Security Groups configured for Jenkins communication

Step 1: Setting Up the Master EC2 Instance

Launch an EC2 Instance (Ubuntu recommended for simplicity):
- Choose Ubuntu AMI.
- Select instance type (e.g., t2.medium).
- Create or select an existing Security Group that allows:
  - HTTP (80)
  - HTTPS (443)
  - SSH (22) for admin access.
Connect to the EC2 instance:
- SSH into the instance using your private key:
```
    ssh -i "your-key.pem" ubuntu@<your-ec2-public-ip>
```

Install Jenkins on the EC2 master instance:

Update the system:

    sudo apt update -y
    sudo apt upgrade -y

Install Java (Jenkins runs on Java):
```
    sudo apt install openjdk-11-jdk -y
```

Install Jenkins:

    wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
    sudo sh -c 'echo deb http://pkg.jenkins.io/debian/ stable main > /etc/apt/sources.list.d/jenkins.list'
    sudo apt update
    sudo apt install jenkins -y

Start Jenkins:
```
    sudo systemctl start jenkins
```
Enable Jenkins to start on boot:
```
    sudo systemctl enable jenkins
```
Open Jenkins web interface:
- Go to http://<your-ec2-public-ip>:8080 in your browser.
- Retrieve Jenkins unlock key by running:
```
    sudo cat /var/lib/jenkins/secrets/initialAdminPassword
```
- Enter this password to unlock Jenkins.

Install required plugins:
- Install plugins for GitHub, Git, Docker, and any other necessary tools from Manage Jenkins → Manage Plugins.
Add SSH Credentials (Private Key) for the Slave EC2 Instance

To add SSH credentials (private key) for the slave EC2 instance in Jenkins, follow these detailed steps:

Step 1: Prepare the SSH Private Key

Before configuring SSH access for the slave EC2 instance, you need to generate an SSH key pair (if you don’t have one already) or use an existing key pair.

1.1 Generate SSH Key Pair

To generate an SSH private and public key pair, use the ssh-keygen command on your local machine or on any Linux/Unix-based system. Follow these steps:
1. Open a Terminal or Command Prompt on your local machine (Mac, Linux, or Windows with Git Bash/WSL).
2. Run the following command to generate the SSH key pair:
```
 bashCopyEditssh-keygen -t rsa -b 2048 -f ~/.ssh/my-ec2-key
```
  Breakdown of the command:
  - -t rsa: Specifies the type of key to generate (RSA).
  - -b 2048: Specifies the number of bits in the key (2048 bits is recommended for RSA).
  - -f ~/.ssh/my-ec2-key: Specifies the file path and name to save the private key (my-ec2-key), typically saved under the .ssh directory.
3. Choose a location to save the key: After running the command, you'll be prompted to enter a file path to save the private key:
```
 bashCopyEditGenerating public/private rsa key pair.
 Enter file in which to save the key (/home/user/.ssh/my-ec2-key):
```
  Press Enter to save it in the default location (or specify a different path if preferred).
4. Enter a passphrase (optional): You will then be prompted to enter a passphrase for the key. This is optional but recommended for extra security. If you choose not to use a passphrase, simply press Enter.
  
  Example output:
```
 bashCopyEditEnter passphrase (empty for no passphrase):
```
5. Key pair generated: After confirming the passphrase (or leaving it empty), the SSH key pair will be generated. You'll have two files:
  - Private Key: my-ec2-key
  - Public Key: my-ec2-key.pub

1.2 View and Copy the Public Key

To view the public key, run the following command:

    bashCopyEditcat ~/.ssh/my-ec2-key.pub

This will display the public key content, which you can then copy.

Step 2: Setting Up the Slave EC2 Instances

Launch additional EC2 instances (for slaves) in the same VPC or subnet as the master instance.
- Use a similar Ubuntu instance.
- Ensure security groups allow inbound SSH access from the master instance.
Install Jenkins slave on the EC2 instance:
- SSH into the slave instance:
```
    ssh -i "your-key.pem" ubuntu@<slave-ec2-public-ip>
```
- Install Java (since Jenkins requires it):
```
    sudo apt install openjdk-11-jdk -y
```
- Install necessary dependencies like git or others for your specific projects.

Step 3: Configuring Jenkins Master and Slave

Add SSH Credentials in Jenkins
1. Login to Jenkins: Open your Jenkins dashboard in a web browser by going to http://<your-jenkins-master-ip>:8080.
2. Navigate to Manage Jenkins:
  - In the Jenkins dashboard, click on Manage Jenkins in the left-hand side menu.
3. Access Manage Credentials:
  - Under the Security section, click on Manage Credentials.
4. Choose the Scope:
  - In the Manage Credentials section, you will see a list of available domains.
  - If you're adding credentials for the global scope (for all Jenkins jobs), click on (global).
5. Add New Credentials:
  - Click on Add Credentials on the left-hand side.
  - In the Kind dropdown, select SSH Username with private key.
6. Fill in the SSH Credentials Form:
  - Username: Enter the SSH username for your slave EC2 instance (typically ubuntu for Ubuntu instances).
  - Private Key:
    - Select Enter directly in the Private Key section.
    - Open your private key file from your master ec2 in a text editor (e.g., vim), and copy the entire contents of the file.
    - Paste the private key contents into the box labeled "Private Key".
  - Passphrase (optional): If your private key is encrypted with a passphrase, enter it here.
  - ID: Provide an identifier (e.g., ec2-slave-key) for the credentials, so it can be easily referenced later.
  - Description: Optionally, provide a description, such as Private Key for connecting to EC2 slave.
7. Save the Credentials:
  - Click OK to save the credentials.

Configure Slave on Jenkins Master:
- Go to Manage Jenkins → Manage Nodes.
- Click on New Node, name it (e.g., slave1), and select Permanent Agent.
- Configure the following:
  - Remote root directory: Path on the slave EC2 instance (e.g., /home/ubuntu/jenkins).
  - Usage: Use this agent for everything.
  - Launch method: Select Launch agent via SSH.
  - Host: Slave EC2's public IP address.
  - Credentials: select credentions which we create named as ec2-slave-key.
Configure Slave public key:
- Navigate to .ssh folder
  - vim /.ssh/authorise_keys
- Open authorise_keys file in edit mode put the public key copied from master EC2’s. and save using esc and put wq! and click Enter.
Verify Slave Connection:
- Jenkins will attempt to connect to the slave using SSH.
- Once successful, the slave will show as connected and ready.

Step 4: Updating Security Groups

Update Security Group of Master EC2:
- Ensure that the master instance’s security group allows inbound communication on the following ports from the slave instance's IP:
  - SSH (22): For agent connection
  - Jenkins port (8080): If you're accessing the Jenkins UI
Update Security Group of Slave EC2:
- Ensure the slave instance’s security group allows inbound SSH (port 22) from the master EC2 instance’s IP.

Step 5: Creating and Managing Jenkins CI/CD Pipelines

Create a Jenkins Pipeline:

In Jenkins, go to New Item → Select Pipeline.

Write the pipeline script (for example, for building a simple Node.js app):

    pipeline {
      agent any

      stages {
        stage('Build') {
          steps {
            sh 'npm install'
          }
        }
        stage('Test') {
          steps {
            sh 'npm test'
          }
        }
        stage('Deploy') {
          steps {
            sh 'deploy.sh'
          }
        }
      }
    }

Run Jobs on the Slave Node:

You can assign specific stages or the entire pipeline to run on the slave node by specifying the node in the pipeline:

    pipeline {
      agent none

      stages {
        stage('Build') {
          agent { label 'slave1' }
          steps {
            sh 'npm install'
          }
        }
        stage('Test') {
          steps {
            sh 'npm test'
          }
        }
        stage('Deploy') {
          agent { label 'slave1' }
          steps {
            sh 'deploy.sh'
          }
        }
      }
    }

Step 6: Testing and Validation

Test the setup:
- Commit code to your repository, triggering the Jenkins pipeline.
- Verify that the pipeline runs on the slave node and passes all stages.
Monitor Jenkins Jobs:
- Check the Jenkins dashboard for job status, logs, and build results.

Step 7: Scaling Jenkins with More Slaves

Add more slaves as needed by following the same process for configuring new EC2 instances and adding them as agents in Jenkins.
Manage Resource Allocation:
- You can dynamically scale the slave nodes by using EC2 Auto Scaling or CloudFormation for automated provisioning.

This is a basic setup for Jenkins CI/CD using AWS EC2 and security groups in a master-slave configuration. It ensures efficient task distribution and builds scalability into your CI/CD pipeline.

Creating and Managing CI/CD Pipelines with Jenkins

Table of contents

Step 1: Setting Up the Master EC2 Instance

1.1 Generate SSH Key Pair