Most developers and platform engineers are all too familiar with the frustrations of CI flows that only run remotely - cumbersome test commits, slow feedback loops, and a generally sluggish and painful experience. But what if there's a better way?

My goal is to enable running the exact same CI flows locally in a cloud-native manner within a development cluster. This approach significantly shortens the feedback loop, boosting productivity and allowing developers to accomplish more in less time.

Introducing the Tools

Tilt

A key tool in this endeavour is Tilt, a dynamic development environment for Kubernetes. Tilt allows you to mirror a real-world setup in your development cluster, offering features like live updates to containers and intelligent rebuilds. While Tilt primarily enhances the development of services and Kubernetes manifests, we'll explore how to extend its functionality to encompass CI processes.

kind

kind (Kubernetes in Docker) is a tool for running local Kubernetes clusters using Docker container nodes. It is primarily designed for testing, CI, and development. kind is lightweight, quick, and is part of the Kubernetes project, ensuring up-to-date support.

ctlptl

ctlptl is a command-line tool for declaratively setting up local Kubernetes clusters for development. By integrating with existing tools such as kind, minikube, etc., it provides a smoother and consistent experience for managing local clusters, focusing on configuration in a way that's friendly to scripting and automation.

Argo Workflows & Events

Argo Workflows is an open-source workflow management system that allows you to orchestrate parallel jobs on Kubernetes. It is designed from the ground up for containers, helping you streamline your pipelines for application development and deployment.

Argo Events is an event-based dependency manager for Kubernetes that helps define, create, and manage derivative actions, chains, or dependencies from events occurring in Kubernetes or elsewhere.

Gitea

Gitea is a lightweight, self-hosted Git service that is easy to install and setup. It's designed to provide a simple and powerful collaboration tool for software development projects, offering features like issue tracking, code reviews, and branch management. Gitea is particularly useful for creating a controlled environment to simulate real-world Git operations such as push events, pull requests, and merges, making it ideal for local CI/CD testing.

How does this all fit together?

In this post, I'll demonstrate how to integrate Tilt with a local kind cluster, enhanced by Argo Workflows & Events, configured as a CI platform. This setup will be supported by an ephemeral Gitea instance to simulate real-world Git operations like push events, pull requests, and more. We'll explore how each tool plays a critical role in creating a powerful local development environment that mirrors your production CI flows without the overhead of remote processing delays.

Let's do it!

Install Tools

First of all, you'll need to install any of the following tools that you don't already have:

Tilt: Automates all the repetitive tasks and lets you focus on the actual development.
kind: Runs local Kubernetes clusters using Docker container nodes.
ctlptl: A command-line tool for setting up local Kubernetes clusters tailored for development environments.
make: Automates the build process in an operating system-agnostic manner.
git: For version control and to clone the repository.
Docker: To containerize applications and run containers.
Kustomize: Manages Kubernetes resources using files.

In addition to this, you'll need a working Go environment. Ensure that Go is properly installed, including setting up your GOPATH and PATH environment variables.

Clone the Repo

I recommend starting by cloning the demo repo for this post; it contains all the code we need to get this working. Here’s the command to clone the repository:

git clone git@github.com:anas-domesticus/cicd-dev-environment.git
cd cicd-dev-environment

Description of Contents

The main elements in this repo we're going to be looking at today are:

Tiltfile.ci: This file contains the Tilt configurations specific for CI processes.
ci-containers/go-build/: This directory includes Dockerfiles and associated scripts for creating the Go build containers we'll use in our example CI flow.
k8s/ci/: Contains Kubernetes workflow templates that we will use with Argo to manage CI/CD pipelines effectively.

These files and directories are crucial for understanding how we build, test, and deploy using our local CI setup. We'll explore each component in detail, explaining how they interact and how they contribute to a more efficient development process.

Let's bring up Tilt

Run the following command in the root of the repo:

make tilt-ci

On the first run, this may take a few moments to pull the relevant containers, but shortly after you should have a prompt inviting you to press the spacebar, do that & a web browser with the Tilt UI should open.

From here, you can click into any resource & be provided with its logs, you also have direct links to both the Gitea & Argo Workflows UIs.

How Things Fit Together

Local Cluster and Tilt Setup:
- The make target initiates a local Kubernetes cluster using kind and launches Tilt.
Tilt Configuration:
- Tiltfile.ci instructs Tilt to install Gitea, Argo Workflows, and Argo Events. Port forwarding is set up automatically for easy access.
Gitea Initialization:
- An initialization script for Gitea is executed. This script sets up a default user with credentials (test-user/password), creates a repository, and configures a webhook for event triggering.
Kustomize Directory Monitoring:
- Tilt actively monitors three Kustomize directories—.tilt/manifests/ci/deps, k8s/ci/webhook, and k8s/ci/workflow-templates. Modifications within these directories are promptly applied to the local cluster.
CI Container Updates:
- Changes in the ci-containers/go-build directory will initiate a Docker build process. Any references to the go-build image in the manifests will automatically be updated to use the newly built container.

Why This is Awesome

This setup emulates a production-like CI environment right on your personal machine! Tilt streamlines the development process by syncing manifests and automatically rebuilding CI containers, offering a rapid feedback loop. Say goodbye to making commits just to test your CI flows!

A Note on Gitea

To circumvent the need for creating commits, the infrastructure surrounding Gitea clones a development repository to a temporary directory on your laptop. Executing the update_gitea script from the Tilt UI will synchronize any changes you have in your actual repository without requiring any code commits within it.

Let's Test It Out

Gitea Access:
- Open your browser and visit http://localhost:3000/test-user/dev. You'll be prompted to log in to access the settings. Use the credentials test-user/password.
Argo Workflows UI:
- Also, visit http://localhost:2746 for the Argo Workflows UI. In the left-hand panel, navigate to "Event Flow" to see a visual representation of the workflow. Below is a diagram for reference.
Webhook Test:
- Now, if you go to http://localhost:3000/test-user/dev/settings/hooks/1, you'll find a "Test Delivery" button. Clicking this button should trigger an example webhook call successfully.
Workflow Execution:
- In the Argo Workflows pane, located at http://localhost:2746/workflows/workflows?&limit=50, you should see a new job titled "go-build". Click on it and open its logs to see the output "I am the CI helper tool".

Congratulations, you've just run your first CI job in the environment!

Now for Something More Interesting...

Open the file ci-containers/go-build/ci-helper/main.go in your preferred editor. The content should look like this:

package main

import "fmt"

func main() {
    fmt.Println("I am the CI helper tool")
}

Obviously, this code isn't particularly complex or useful; it's just for demonstration purposes. Change the print line to a message of your choice.

After saving the file, watch your Tilt UI. You should notice the go-build resource rebuilding. Once it's finished and the status turns green, repeat steps 3 and 4 from the "Let's Test It Out" section. This time, you should see your custom log line in the output.

Notice how swiftly you can iterate on your CI containers!

What About the Jobs Themselves?

Let's open up k8s/ci/workflow-templates/go-build.yaml and change what gets run inside the container. Update the file as shown below:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: go-build
  namespace: workflows
spec:
  serviceAccountName: argo-workflow
  entrypoint: go-build
  templates:
  - name: go-build
    inputs:
      parameters:
      - name: message
    container:
      image: go-build
      command: ["/usr/bin/echo"]
      args: ["Hello!"]

Now, if we repeat the test delivery in Gitea and check the latest Workflow run, we'll see that instead of running the Go binary baked into the container, we're simply echoing out a friendly greeting.

What next?

Hopefully, from this post, you're able to see the value in this fast-feedback way of working. Of course, this is a demo repository, so it has its limitations.

In a real-world scenario, you would need to develop actually useful CI jobs and likely create Kustomize overlays to make this codebase functional both in development and production environments.

I have been working on a CI helper binary+container for this setup to allow webhook schemas from mainstream git repositories (like GitHub, etc.) to be parsed and standardised for a truly agnostic experience. I will update this post with a link when it's ready!

Happy coding!

Revolutionising CI with Argo Events, Workflows & Tilt: Faster, Real-Time Developer Feedback