Observability:

Observability is a critical aspect of modern software systems, ensuring that you can understand what’s happening inside a system based on its outputs. The key pillars of observability are:

• 1. Monitoring

  • Purpose: Continuously observe the health and performance of applications, infrastructure, and services.

  • Key Metrics: CPU usage, memory utilization, response times, error rates, etc.

  • Tools: Prometheus, Grafana, Datadog, New Relic.

2. Logging

• Purpose: Capture detailed records of events and state changes in a system for debugging and analysis.

• Types of Logs: Application logs, system logs, security logs, audit logs.

• Best Practices: Structured logging, centralized log management, log rotation.

• Tools: ELK Stack (Elasticsearch, Logstash, Kibana), Fluentd, Splunk.

3. Tracing

• Purpose: Track the flow of requests across services to understand dependencies and pinpoint performance bottlenecks.

• Key Concepts: Spans, traces, and context propagation.

• Use Case: Identifying slow microservices in a distributed architecture.

• Tools: Jaeger, Zipkin, OpenTelemetry.

4. Alerting

• Purpose: Notify relevant teams when predefined conditions indicate potential issues or outages.

• Alert Types: Threshold-based alerts, anomaly detection, multi-condition alerts.

• Best Practices: Avoid alert fatigue, prioritize actionable alerts, and integrate with incident response workflows.

• Tools: PagerDuty, Opsgenie, Prometheus Alertmanager.

Hands-on work

1. Create AWS EC2 instance (t2.medium(Ubuntu)) with at least a 15 GB SSD because a lot of tools will install in this Linux.

2. Install docker. Run the command below:

sudo apt-get install docker.io -y
#give permission as a user
sudo usermod -aG docker $USER && newgrp docker

3. Now clone repo to ec2 instance

   link: https://github.com/mmohamm5/Observability-by-using-K8s_Kind-Prometheus-Grafana_Helm_chart.git

git clone https://github.com/mmohamm5/Observability-by-using-K8s_Kind-Prometheus-Grafana_Helm_chart.git

3. Now go to the cloned repo folder. then go to the “kind-cluster” folder. Here you will see config files YAML files. and shell script to install the kid K8s-kind.

config.yml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4

nodes:
- role: control-plane
  image: kindest/node:v1.30.0
- role: worker
  image: kindest/node:v1.30.0
- role: worker
  image: kindest/node:v1.30.0

This file will create three nodes. One for the control plane and two worker nodes. Kind will make a Docker container of control plane and worker node.

4. Now install k8s-kind by using “install_kind.sh” shell script

#!/bin/bash

# For AMD64 / x86_64

[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
chmod +x ./kind
sudo cp ./kind /usr/local/bin/kind
rm -rf kind

Note: Give executive permission to execute this shell script.

chmod +x install_kind.sh

Then execute the shell.

./install_kind.sh

5. Now install Kubectl to communicate with the cluster. This is the shell script to install Kubectl in this cluster. In my repo file name is “install_kubectl.sh”

#!/bin/bash

# Variables
VERSION="v1.30.0"
URL="https://dl.k8s.io/release/${VERSION}/bin/linux/amd64/kubectl"
INSTALL_DIR="/usr/local/bin"

# Download and install kubectl
curl -LO "$URL"
chmod +x kubectl
sudo mv kubectl $INSTALL_DIR/
kubectl version --client

# Clean up
rm -f kubectl

echo "kubectl installation complete."

Note: Give permission to this file for execution and execute this as like previous steps.

6. Now create a kind cluster and run the following commands.

kind create cluster --config=config.yml --name=my-cluster
or
kind create cluster --config=config.yml
#Check cluster informations
kubectl cluster-info --context kind-kind
kubectl get nodes
kind get clusters

Output will look like this:

To see the pods run the following command:

kubectl get pods -A

7. Once a cluster is created, we will deploy our application. For this, we have two options: one through ArgoCD or directly by using the k8s manifest file.

   Now we will see K8S deployment. For this k8s deployment, we go to the “k8s-specifications” folder, which is from github clone. There we will see a lot of k8s manifest files like deployment, service, and so on.

   Simply here, run the following command to run all of this manifest file together.

    kubectl apply -f .
   

   Once it's done, we check like this:

Till now, we have set our environment and deployed our applications. Now we will do observability.

Note: Make sure your necessary ports are open (8080,9090, 31002,31000,9090,32000,32001,)

We will use Prometheus for collecting metrics. Prometheus basically stores the data as a time series data base. From this data, we will make a graph.

HELM

HELM is a package manager for the k8s manifest file where you can install, manage, delete, uninstall, and a lot of repositories for any applications that are to be deployed on k8s.

We will use HELM to install Prometheus and Grafana.

Install HELM

Run the following command.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Once it is done, we will download a shell script. “get_helm.sh”

Now our helmis is installed .

1. Install Prometheus and Grafana by using helm charts.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add stable https://charts.helm.sh/stable
helm repo update
kubectl create namespace monitoring
helm install kind-prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --set prometheus.service.nodePort=30000 --set prometheus.service.type=NodePort --set grafana.service.nodePort=31000 --set grafana.service.type=NodePort --set alertmanager.service.nodePort=32000 --set alertmanager.service.type=NodePort --set prometheus-node-exporter.service.nodePort=32001 --set prometheus-node-exporter.service.type=NodePort
kubectl get svc -n monitoring
kubectl get namespace

Output:

Output:

We can see Prometheus with specific ports. Now we will forward this port.

kubectl port-forward svc/kind-prometheus-kube-prome-prometheus -n monitoring 9090:9090 --address=0.0.0.0 &

For Prometheus, expose port 9090 & copy EC2 ‘s public IP and paste it to a new tab with the 9090 port.

Now in the Prometheus query tab, you will ´give any kind of expression. Like this:

#To see cpu usages , with default name spaces. 
sum (rate (container_cpu_usage_seconds_total{namespace="default"}[1m])) / sum (machine_cpu_cores) * 100

#To see container _memory usages
sum (container_memory_usage_bytes{namespace="default"}) by (pod)


sum(rate(container_network_receive_bytes_total{namespace="default"}[5m])) by (pod)
sum(rate(container_network_transmit_bytes_total{namespace="default"}[5m])) by (pod)

Now see the query in a graph.

We can see the last 5 minutes of our CPU usage in this graph.

Now see pods network

Now we will run our APP (vote) then we will see how our cpu usages.

Now forward the port of this app

kubectl port-forward svc/vote 5000:5000 --address=0.0.0.0 &

Note: Expose this port to your EC2 instance. Now take the public IP of the instance and open the tab.

do something with this app. like new tab. vote, reload.

We can see our CPU usages increased.

Till now we can see all queries part by part, but we need to see all query metrics together in a graphical presentation. For this we will use Grafana.

2. We already made pods of grafana by using helm charts.

   Now just forward the ports to get access from the the browser.

kubectl port-forward svc/kind-prometheus-grafana -n monitoring 3000:80 --address=0.0.0.0 &

Note: expose 3000 ports into your EC2 instance security group.

Added data sources and created dashboard.

Create a new user:

Let’s say you dont want to give access to your dashboard to others but give the permission only for view then follow these steps.

Now I can share my dashboard to others.

Firstly, share the URL or public IP with others and give the user name and password. which we created as users

My case I used: user name: demo & password: demo

They will get the access, but as a user. only for view.

Now connections in Grafana.

As we deployed it through HELM chats at that time, data sources were already added. See the photo below.

Now click Build a dashboard

Now click add Vizualizations.

select data sources and see data sources added.

Once we run the query, we will see visualizations.

We can choose which type of dashboard we want to see.

I chose a bar chart, and the time frame is 15 minutes, and save it. Once it is saved, we can make other visualization dashboards.

I took another dashboard, & this time the container network received packets.

Now we will see k8s dashboard. for the go to Google and search Grafana dashboard.

link: https://grafana.com/grafana/dashboards/

We can choose templates from here. Choose any one. In my case, I choose Kubernetes. I search it into the search bar as a kubenetes.

Once you click on this, you will see details about this dashboard. Now we will import this dashboard to our Grafana. For this, scroll down a little bit and see that on the right side you will see Dashboard ID. Just copy it.

Once you copy the dashboard ID, now go to your Grafana dashboard > new dashboard > import > paste your dashboard ID.

Now click load. Then you will see, like this:

Select the default source and import it.

Now we will see our dashboard is imported.

THE END