📌 Advanced Kubernetes Monitoring with Prometheus, Grafana, Node Exporter & cAdvisor

1️⃣ Overview

Monitoring Kubernetes effectively requires collecting metrics from both the cluster and individual nodes. This guide extends our monitoring setup by adding:

✅ Prometheus – Collects Kubernetes cluster metrics
✅ Grafana – Visualizes metrics in dashboards
✅ Node Exporter – Captures node-level CPU, memory, and disk metrics
✅ cAdvisor – Monitors container-level resource usage

By the end of this guide, you'll have deep visibility into your Kubernetes environment. 🚀

---

2️⃣ Deploying Prometheus

📌 Ensure you have created a monitoring namespace:

kubectl create namespace monitoring

✅ Sample Output:

namespace/monitoring created

🔹 Apply Prometheus Configurations

curl -o prometheus-config.yaml https://raw.githubusercontent.com/Vikas-DevOpsPractice/EasyShop/feature/kindcluster/K8s/14-prometheus-config.yaml
curl -o prometheus-deployment.yaml https://raw.githubusercontent.com/Vikas-DevOpsPractice/EasyShop/feature/kindcluster/K8s/15-prometheus-deployment.yaml

kubectl apply -f prometheus-config.yaml -n monitoring
kubectl apply -f prometheus-deployment.yaml -n monitoring

✅ Check Prometheus Deployment:

kubectl get pods -n monitoring

✅ Sample Output:

NAME                         READY   STATUS    RESTARTS   AGE
prometheus-5f9d77c86f-xyz12  1/1     Running   0          1m

---

3️⃣ Deploying Node Exporter

🔹 Why Node Exporter?

🔹 Collects CPU, Memory, Disk, and Network usage of each Kubernetes node
🔹 Provides hardware and OS metrics

🔹 Step 1: Create a DaemonSet for Node Exporter

# node-exporter-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.5.0
        ports:
        - containerPort: 9100
          hostPort: 9100

✅ Apply the DaemonSet:

kubectl apply -f node-exporter-daemonset.yaml -n monitoring

✅ Verify Deployment:

kubectl get pods -n monitoring

✅ Sample Output:

NAME                         READY   STATUS    RESTARTS   AGE
node-exporter-xyz12          1/1     Running   0          1m

---

4️⃣ Deploying cAdvisor

🔹 Why cAdvisor?

🔹 Provides per-container resource usage (CPU, memory, disk, network)
🔹 Helps in troubleshooting slow or resource-hungry containers

🔹 Step 1: Create a DaemonSet for cAdvisor

# cadvisor-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cadvisor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: cadvisor
  template:
    metadata:
      labels:
        app: cadvisor
    spec:
      hostNetwork: true
      containers:
      - name: cadvisor
        image: gcr.io/cadvisor/cadvisor:v0.47.0
        ports:
        - containerPort: 8080
          hostPort: 8080

✅ Apply the DaemonSet:

kubectl apply -f cadvisor-daemonset.yaml -n monitoring

✅ Verify Deployment:

kubectl get pods -n monitoring

✅ Sample Output:

NAME                         READY   STATUS    RESTARTS   AGE
cadvisor-xyz12               1/1     Running   0          1m

---

5️⃣ Integrating Node Exporter & cAdvisor with Prometheus

🔹 Update Prometheus Configuration

Add the following scrape jobs to prometheus-config.yaml:

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter.monitoring.svc.cluster.local:9100']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor.monitoring.svc.cluster.local:8080']

✅ Reapply Prometheus Configuration:

kubectl apply -f prometheus-config.yaml -n monitoring
kubectl rollout restart deployment prometheus -n monitoring

---

6️⃣ Deploying Grafana

curl -o grafana-deployment.yaml https://raw.githubusercontent.com/Vikas-DevOpsPractice/EasyShop/feature/kindcluster/K8s/16-grafana-deployment.yaml
kubectl apply -f grafana-deployment.yaml -n monitoring

✅ Verify Grafana:

kubectl get pods -n monitoring

✅ Sample Output:

NAME                         READY   STATUS    RESTARTS   AGE
grafana-78b6c9c76f-xyz12     1/1     Running   0          1m

---

7️⃣ Setting Up Dashboards in Grafana

🔹 Add Prometheus as a Data Source

📌 Go to Grafana → Configuration → Add Data Source
🔹 Select Prometheus
🔹 Set URL to:

http://prometheus.monitoring.svc.cluster.local:9090

🔹 Click Save & Test

✅ Integration Successful!

---

🔹 Import Prebuilt Kubernetes Dashboards

📌 Go to Grafana Dashboard → Click Dashboards → Import
🔹 Use Dashboard ID: 11074 (Node Exporter)
🔹 Use Dashboard ID: 13689 (cAdvisor)
🔹 Select Prometheus as the data source → Click Import

✅ Sample Node Metrics Dashboard:

✅ Sample Container Metrics Dashboard:

---

8️⃣ Setting Up Alerts in Grafana

📌 Open Grafana → Click Alerts → Create Alert Rule
🔹 Condition: Alert when CPU Usage > 80% for 5 minutes
🔹 Notification: Email, Slack, PagerDuty
🔹 Click Save & Enable Alerting

✅ Now, alerts will trigger on resource spikes!

---

9️⃣ Troubleshooting & Best Practices

🔹 Prometheus Not Collecting Metrics?

kubectl logs -l app=prometheus -n monitoring

📌 Ensure the scrape configs in prometheus-config.yaml are correct

---

🔹 Node Exporter Not Running?

kubectl describe pod node-exporter-xyz12 -n monitoring

📌 Ensure hostPort 9100 is available

---

🔹 cAdvisor Metrics Not Appearing?

kubectl logs -l app=cadvisor -n monitoring

📌 Ensure cadvisor.monitoring.svc.cluster.local:8080 is correct in Prometheus config

---

🎯 Conclusion

🚀 Advanced Kubernetes Monitoring is now fully set up!
✅ Prometheus collects Kubernetes & node metrics
✅ Node Exporter tracks hardware performance
✅ cAdvisor monitors per-container usage
✅ Grafana visualizes & alerts on key metrics

📌 Next Step: End-to-End CI/CD Automation for Kubernetes Using Jenkins, GitLab, AWS CodePipeline & Azure DevOps