This blog post guides you through a chaos engineering experiment on a Kubernetes-deployed application using Chaos Mesh. We will also monitor our application's performance using Prometheus and Grafana.

Prerequisites

Kubernetes cluster
Helm installed

Steps

1. Install Prometheus using Helm

First, add the Prometheus Helm chart repository and update your Helm repo.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Next, install Prometheus.

helm install my-prometheus prometheus-community/prometheus

2. Configure Prometheus

Create a values.yaml file with the following configuration to scrape metrics from specific pods:

serverFiles:
  prometheus.yml:
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: (vote|redis|worker|result|db)

Apply the configuration:

helm upgrade my-prometheus prometheus-community/prometheus -f values.yaml -n default

3. Install Grafana

Add the Grafana Helm chart repository and update your Helm repo.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Next, install Grafana.

helm install my-grafana grafana/grafana

Retrieve the Grafana admin password:

kubectl get secret --namespace default my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Port-forward Grafana to access it locally:

kubectl port-forward service/my-grafana 3000:80 --address="0.0.0.0"

4. Configure Grafana

Open Grafana at http://localhost:3000 and log in with the admin credentials.
Add Prometheus as a data source with the URL http://my-prometheus-server.
Import any dashboard of your liking from the plethora of dashboards available. For example, the Kubernetes cluster monitoring dashboard with ID 7824.

5. Deploy the Sample Kubernetes Application

Clone the example voting app:

git clone https://github.com/dockersamples/example-voting-app.git

Deploy the application to your Kubernetes cluster:

kubectl create -f example-voting-app/k8s-specifications/

6. Install Chaos Mesh

Add the Chaos Mesh Helm chart repository and update your Helm repo.

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update

Install Chaos Mesh:

helm install chaos-mesh chaos-mesh/chaos-mesh

7. Create and Apply a Chaos Experiment

Create a pod-kill.yaml file with the following content:

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-example
  namespace: default
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - default
    labelSelectors:
      app: vote

Apply the chaos experiment:

kubectl apply -f pod-kill.yaml

The vote pod will be killed, and a new pod will be created. One can login into Grafana and view dashboards.

Conclusion

In this simple chaos experiment, we successfully killed a pod using Chaos Mesh, demonstrating the resilience of Kubernetes. When the pod was killed, Kubernetes immediately took action to create a new pod, ensuring the application remained available and continued to function properly.

This experiment highlights the robustness of Kubernetes' self-healing capabilities and provides a glimpse into the potential of chaos engineering. For those looking to dive deeper, Chaos Mesh offers a user-friendly dashboard where more advanced chaos experiments can be designed and monitored.

Useful Links

Prometheus:https://prometheus.io/
Grafana:https://grafana.com/
Chaos Mesh:https://chaos-mesh.org/

These tools provide powerful capabilities for monitoring, visualizing, and testing the resilience of your applications. Explore them to enhance your observability and chaos engineering practices.

Chaos Engineering Experiment with Kubernetes and Chaos Mesh

Table of contents