Self-Hosting Runners for GitHub Actions: A Complete Tutorial

🚀 Project Overview

This article provides a comprehensive guide to the GitHub Actions Self-Hosted ECR Image project, detailing each component, the challenges encountered, and the solutions implemented. The primary goal is to offer readers a thorough understanding of the project's structure and functionality.

• This exercise was performed to automate the deployment of a simple Python application using GitHub Actions in conjunction with AWS Elastic Container Registry (ECR).

• Leveraging self-hosted runners enhances the CI/CD processes, offering greater control and customization over the environment compared to GitHub’s hosted runners

• This setup allows for the management of containerized workflows and facilitates seamless deployment to container registries like AWS ECR.

• Throughout this guide, we will break down the setup process, simplify its components, and provide step-by-step instructions on how to replicate this setup for your own projects.

• This includes integrating GitHub Actions with self-hosted runners on any cloud provider, focusing on efficient container management and deployment strategies.

📜 Problem Statement

There has always been a need for a streamlined CI/CD process for deploying applications, but several challenges emerge in managing dependencies and ensuring reproducibility without self-hosted runners:

• Limited Customization: Default GitHub-hosted runners offer limited customization options, restricting the ability to tailor the CI/CD environment to specific project requirements, which can hinder the development process.

• Resource Constraints: Default hosted runners may not provide the necessary resources for large or resource-intensive builds, leading to slower build times and potential bottlenecks in the CI/CD pipeline.

• Dependency Management: Managing dependencies can be challenging due to the lack of control over the runner environment, which can lead to inconsistencies and difficulties in ensuring reproducibility across different builds.

• Data Security Concerns: Using public runners may raise data security concerns, as sensitive data and proprietary code are processed on shared infrastructure, potentially exposing them to security vulnerabilities.

• Network Access Limitations: Hosted runners may not have access to private networks or internal resources, which can be a significant limitation for projects that require integration with internal systems or databases.

• Cost Implications: For organizations with high usage, relying solely on GitHub-hosted runners can lead to increased costs due to the consumption of paid runner minutes, especially for open-source projects or those with frequent builds.

• Scalability Issues: Scaling the CI/CD process can be challenging with hosted runners, as organizations are limited by the availability and capacity of GitHub's infrastructure, potentially impacting the ability to handle increased workloads efficiently.

🔧 Solutions Implemented

Before diving in, let’s understand the motivation behind this setup:

• Self-Hosted Runners: Unlike GitHub’s default hosted runners, self-hosted runners provide you with full control over the CI/CD environment. This allows running on our own infrastructure, such as AWS EC2, enabling customization of hardware and access to private resources like AWS ECR without relying on public runners.

• GitHub Actions: Utilized GitHub Actions to automate the build and deployment process, streamlining the workflow and enhancing efficiency.

• Cost Reduction and Control: By using self-hosted runners, the setup reduces costs associated with GitHub-hosted runner minutes and increases control over the CI/CD environment, allowing for tailored configurations and optimizations.

🌟 Features

• Seamless Integration: Effortlessly connect GitHub Actions with self-hosted runners across any cloud environment, ensuring smooth and efficient CI/CD processes.

• Container Management: Utilize Docker to build, test, and deploy containerized applications, streamlining the development and deployment lifecycle.

• AWS ECR Deployment: Leverage AWS Elastic Container Registry as a secure and scalable container registry for storing Docker images. It integrates seamlessly with AWS services, ensuring our images remain private and accessible within our VPC, while automating deployments to AWS ECR.

• Scalable and Cost-Effective: Implement cloud-based self-hosted runners to execute workflows efficiently, optimizing resource usage and reducing costs.

• Customizable: Fully configure the setup to accommodate diverse CI/CD pipelines, allowing for tailored solutions that meet specific project requirements.

🚀 Getting Started

📋 Prerequisites

1. Create an IAM Role for EC2 Instance:

   • Establish an IAM role with the necessary permissions for the EC2 instance to interact with AWS services like ECR.

   • Attach this role to the EC2 instance to allow secure and managed access without embedding credentials.

2. Set Up the AWS ECR Repository: We need a place to store our Docker images. AWS ECR can be ideal for this.

   • Navigate to the AWS Management Console.

   • Go to ECR > Repositories > Create Repository.

   • Name the repository (e.g.github-runneror your preferred name) and create it to store your Docker images.

3. Launch an EC2 instance with Docker installed and AWS CLI configured.

   • Set up your AWS credentials to enable connections to AWS ECR.

   • Spin up an EC2 instance (e.g., t2.medium with Ubuntu 24.04) suitable for the workload.

   • Install Docker and AWS CLI on the instance.

   • Ensure the instance has internet access and the security group allows necessary outbound connections.

4. Configure AWS Credentials:

   • Preferred Method: Utilize the IAM role attached to the EC2 instance for seamless and secure access to AWS services.

   • Alternative Methods:

     • Use an AWS credentials file located at ~/.aws/credentials.

     • Set environment variables:

         export AWS_ACCESS_KEY_ID=your-access-key
         export AWS_SECRET_ACCESS_KEY=your-secret-key
       

     • Note: Avoid hardcoding credentials; prefer IAM roles or AWS credentials files for enhanced security.

5. A GitHub Actions workflow to build, tag, and push images to ECR.

6. Set Up Self-Hosted Runners: Install and register a self-hosted runner for the GitHub repository. Learn more from the official documentation on how to set up the runner.

   The GitHub self-hosted runner must be running and listening for jobs when the pipeline runs.

7. Once done, there will be a run.sh script available which will help start the runner using the command:

   Ensure this command is kept running in the terminal or as a background process before triggering the workflow.

8. Add an inbound traffic rule in the EC2 instance security group to allow traffic from the port (in this case is 8080) where the application runs.

9. Run the application as a docker container: Firstly,

   • Authenticate Docker to the ECR registry.

   • Pull the image from ECR

   • Run the image as a container

   • Evaluate docker logs for the app events (for debugging)

Follow the steps specified in Pulling and Running the ECR Image on EC2 instance section of theREADME.md file (see Documentation below)

---

📂 Project Structure

├── .github/              # GitHub configuration and workflows
│   └── workflows/        # GitHub Actions workflow files
├── docker/               # Dockerfiles and related resources
├── test.py/              # Test case for ensuring workflow reliability
├── images/               # Image assets for documentation or usage
├── Dockerfile            # Main Docker setup
├── LICENSE               # License for the project
├── pyproject.toml        # Python project configuration
├── README.md             # Project overview and instructions
├── uv.lock               # Python dependency lock file
├── .dockerignore         # Docker ignore rules
├── .gitignore            # Git ignore rules
├── .python-version       # Python version specification

---

⚙️ GitHub Actions Workflow

Below is an example workflow file (.github/workflows/deploy.yml) for automating deployments to AWS ECR:

name: Build & Deploy Python App to AWS ECR repository

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]
  workflow_dispatch:

env:
  AWS_REGION: ${{ vars.AWS_REGION }}
  ECR_REPOSITORY: ${{ vars.ECR_REPOSITORY }}

jobs:
  install:
    name: Install uv & other dependencies
    runs-on: self-hosted

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version-file: "pyproject.toml"

    - name: Install uv
      uses: astral-sh/setup-uv@v5

    - name: Enable caching
      uses: astral-sh/setup-uv@v5
      with:
        enable-cache: true

    - name: Install project dependencies
      run: uv sync --locked --all-extras --dev

    - name: Run tests with pytest
      run: uv run pytest -vs test_calculator.py

  build:
    name: Build & push docker image
    needs: install
    runs-on: self-hosted

    steps:
    - uses: actions/checkout@v4    

    - name: Configure AWS credentials
      if: success()
      uses: aws-actions/configure-aws-credentials@v4
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR repository
      if: success()
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v2

    - name: Build, tag, and push image to ECR
      if: success()
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        ECR_REPOSITORY: ${{ env.ECR_REPOSITORY }}
      run: |
        SHORT_SHA=$(echo $GITHUB_SHA | tail -c 6)
        TAG_DATE=$(date +"%d-%b-%y")
        BRANCH_NAME=$(echo ${GITHUB_REF_NAME} | sed 's/\//-/g')
        IMAGE_TAG="${BRANCH_NAME}-${SHORT_SHA}-${TAG_DATE}"
        docker build . --file Dockerfile --tag $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

---

🧪 Testing

Unit tests are performed using the uv package manager and the pytest library using the following command:

uv run pytest -vs <file-name.py>

---

Finally, the application is served using FastAPI, which serves as the web framework, and Uvicorn, which acts as the ASGI server.

When the Docker container is executed, it initiates the FastAPI application, making it accessible on port 8080

This setup allows the application to handle incoming HTTP requests efficiently, leveraging FastAPI's capabilities for building APIs and Uvicorn's performance as a lightweight and fast server.

---

📊 Visualizations

Figure: Self hosted runner in active state listening for connections

Figure: EC2 instance self hosted runner deployment successful.

Figure: Uvicorn powered FastAPI app running on EC2 instance self hosted runner

📖 Documentation

Detailed documentation is available in my github repository(linked under), including few links:

• Setup Guide: Step-by-step instructions to configure the project are specified in the README.md file in the github project: gh-actions-self-hosted-ecr

• Best Practices: Pro-Tips for optimizing workflows (refer Security Best Practices section of the same README.md file)

🔗 Useful Links

• GitHub Actions Documentation

• AWS Elastic Container Registry

• Self-Hosted Runners

• Installation guide for uv package manager

  • Using uv in Docker

🤝 Contributing

Contributions are welcome! To contribute:

1. Fork the repository.

2. Create a new branch (git checkout -b feature/your-feature).

3. Commit your changes (git commit -m "Add your feature").

4. Push the branch (git push origin feature/your-feature).

5. Open a Pull Request.

Please follow the Code of Conduct.

---

Conclusion

• This project demonstrates the integration of GitHub Actions with AWS ECR & EC2 instances for deploying Python applications using uv as package manager.

• Using self-hosted runners provides flexibility and cost savings.

---

❤️ Acknowledgments

Special thanks to the open-source community for providing tools and resources that made this exercise possible.