Introduction

Prometheus is widely used for monitoring and alerting in cloud-native environments, particularly in Kubernetes. When collecting metrics, they can generally be categorized into three levels:

• Node-Level Metrics (System-level metrics)

• Cluster-Level Metrics (Kubernetes-level metrics)

• Application-Level Metrics (Application-specific metrics)

Understanding these levels helps in designing an efficient monitoring setup that provides insights into infrastructure health, cluster performance, and application behavior.

1. Node-Level Metrics

What Are Node-Level Metrics?

Node-level metrics provide insights into the health and performance of the underlying infrastructure (VMs, physical machines, or Kubernetes nodes). These metrics are essential for understanding system resource utilization and detecting performance bottlenecks.

Examples of Node-Level Metrics:

How to Collect Node-Level Metrics?

The most commonly used exporter for collecting node-level metrics is Node Exporter.

Install Node Exporter on a Linux Node:

wget https://github.com/prometheus/node_exporter/releases/latest/download/node_exporter-linux-amd64.tar.gz
tar -xvzf node_exporter-linux-amd64.tar.gz
./node_exporter &

By default, Node Exporter exposes metrics at http://localhost:9100/metrics, which Prometheus can scrape.

Configure Prometheus to Scrape Node Exporter:

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

2. Cluster-Level Metrics

What Are Cluster-Level Metrics?

Cluster-level metrics provide insights into Kubernetes resource utilization. These metrics help in monitoring the health and performance of the Kubernetes cluster, including nodes, pods, deployments, and API server interactions.

Examples of Cluster-Level Metrics:

How to Collect Cluster-Level Metrics?

For Kubernetes environments, the Kube State Metrics exporter is commonly used to gather cluster-level metrics.

Deploy Kube State Metrics in Kubernetes:

kubectl apply -f https://github.com/kubernetes/kube-state-metrics/releases/latest/download/kube-state-metrics.yaml

This exposes Kubernetes resource metrics at /metrics, which Prometheus can scrape.

Configure Prometheus to Scrape Kube State Metrics:

scrape_configs:
  - job_name: "kube-state-metrics"
    static_configs:
      - targets: ["kube-state-metrics.kube-system.svc.cluster.local:8080"]

3. Application-Level Metrics

What Are Application-Level Metrics?

Application-level metrics provide insights into the performance and behavior of individual applications running inside the cluster. These metrics can include HTTP request counts, response times, error rates, database queries, and business-specific KPIs.

Examples of Application-Level Metrics:

How to Collect Application-Level Metrics?

Applications must expose their own metrics using instrumentation libraries or exporters.

Example: Exposing Metrics in a Python Flask App

from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app)

@app.route('/')
def home():
    return "Hello, World!"

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

This exposes metrics at http://localhost:8080/metrics.

Configure Prometheus to Scrape the Application Metrics:

scrape_configs:
  - job_name: "flask-app"
    static_configs:
      - targets: ["localhost:8080"]

Summary

---

Conclusion

Understanding Node-Level, Cluster-Level, and Application-Level Metrics is crucial for effectively monitoring a Kubernetes-based system.

• Node-level metrics help in tracking infrastructure health.

• Cluster-level metrics give insights into Kubernetes resources and workloads.

• Application-level metrics monitor app performance and business KPIs.

Each level requires the right exporters and Prometheus configurations to ensure a comprehensive monitoring setup.