Setting Up Prometheus & Grafana for Kubernetes Monitoring

Hey folks! Back with a fresh blog.

These days, monitoring is literally buzzing in the world of DevOps. And you might ask, “Do I really need to learn this?”
Yes! It’s a skill that can seriously boost your resume.

Today, we’ll learn how to set up Prometheus and Grafana in Kubernetes. And here’s a pro tip for newbies: you don’t need to build dashboards from scratch. Using a template ID, you can get dashboards with commonly used queries in seconds.

So, let’s dive in!

Why Monitoring?

Imagine your organization has one Kubernetes cluster. Monitoring it is manageable.

But what if you have multiple projects running? Monitoring everything manually becomes a nightmare.

Here’s where Prometheus comes in:

• It collects metrics exposed by the Kubernetes API server.

• You can monitor resources efficiently across all clusters.

Prometheus Architecture

Here’s how it works:

• Prometheus server scrapes metrics from targets using HTTP.

• Metrics are stored in a time-series database on disk (HDD/SSD).

• Alertmanager can send notifications to Slack, Email, etc.

And then we have Grafana:

• Grafana lets us visualize metrics using charts, graphs, and dashboards.

Step 1: Install Prometheus Using Helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus

Check pods:

kubectl get pods

Check services:

kubectl get svc

You’ll see prometheus-server and kube-state-metrics running as ClusterIP.

Tip: Kube-state metrics give extra insights beyond the API server.

Step 2: Expose Prometheus & Kube-State Metrics as NodePort

kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=pro-ser
kubectl expose service prometheus-kube-state-metrics --type=NodePort --target-port=8080 --name=kube-state

Check services again:

kubectl get svc

Access Prometheus UI:

http://<worker-node-ip>:<nodeport>

Step 3: Install Grafana

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install grafana grafana/grafana

Get credentials:

kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode

• User: admin

• Password: output from command

Expose Grafana:

kubectl expose service grafana --type=NodePort --name=grafana-nodeport --target-port=3000

Login: <worker-node-ip>:<nodeport>

Step 4: Connect Grafana to Prometheus

1. Go to Data Sources → Add Data Source

2. Select Prometheus

3. Enter Prometheus URL → Save & Test

Step 5: Import a Dashboard

Instead of creating a dashboard from scratch, here’s a shortcut:

1. Click Dashboard → Import.

2. Enter the ID 3662 (Grafana’s prebuilt template with commonly used queries).

3. Save, and your dashboard is ready!

Step 6: Explore Metrics with Kube-State

To monitor replica counts, pods, and other desired states:

To get metric endpoint

type:<worker-node-ip>:<nodeport>,nodeport which binded when you exposed kube-state-metrics as NodePort.

1. Go to Prometheus UI:

2. Explore metrics and queries.

3. You can execute queries and see JSON outputs.

Pro Tip: Add custom jobs in Prometheus ConfigMap:

kubectl edit cm prometheus-server

• Under scrape_configs, add job_name & target port

• Save & reload

Your custom metrics are now being scraped!

Congrats! You now have a working Prometheus + Grafana setup in Kubernetes.

Hope you like it :)