Day 42 of 90 Days of DevOps Challenge: Creating Grafana Dashboards with Prometheus

On Day 41, I explored PromQL and how Prometheus becomes a proactive monitoring system with well-defined alert rules.

Today, it’s all about visualization, because raw metrics are great, but human-readable dashboards are even better. In this blog, I’ll take you through how to build powerful, dynamic, and visually rich Grafana dashboards to visualize data from Prometheus.

Getting Started with Grafana

Grafana is an open-source analytics and monitoring solution that supports multiple data sources; however, our focus today is on Prometheus.

Access Grafana

If you're using Kubernetes with Prometheus Operator:

• Run kubectl get svc -n monitoring to find the LoadBalancer IP or NodePort of your Grafana service.

• Open Grafana in your browser using that IP:
  http://<LoadBalancer_IP>:<Port>

Log in to Grafana

• Username: admin

• Password: prom-operator (or set via secret/config map)

Configure Prometheus as a Data Source

1. Navigate to "Configuration" > "Data Sources"

2. Click “Add data source.”

3. Select Prometheus

4. Enter the Prometheus URL:

   • For local setups: http://localhost:9090

   • For Kubernetes (via kube-prometheus-stack):
     http://prometheus-k8s.monitoring.svc.cluster.local:9090

5. Click Save & Test
   Grafana will validate and confirm the connection.

Building Dashboards and Panels

A dashboard is a collection of panels that provide a holistic view of your infrastructure.

Here’s how to take it further:

• Add multiple panels to monitor different metrics (CPU, memory, disk I/O, network usage, etc.)

• Use repeating panels to generate graphs per pod, container, or node dynamically

• Import pre-built dashboards from Grafana Labs Dashboards

Create a New Dashboard

1. Click the "+" icon in the sidebar → Select Dashboard

2. Click "Add new panel" to begin visualizing your first metric

Example Panels Using PromQL

CPU Usage Panel

PromQL Query:

rate(container_cpu_usage_seconds_total[5m])

• Shows the per-second average rate of CPU usage over the past 5 minutes.

• You can break it down further by container or namespace using sum by(pod) or container.

Memory Availability Panel

PromQL Query:

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100

• Shows percentage of memory available.

• Multiply by 100 to show as percentage.

Network I/O

rate(container_network_receive_bytes_total[5m])

• Useful for identifying bandwidth bottlenecks or packet floods.

Disk I/O

rate(container_fs_reads_bytes_total[5m])

Organize Your Dashboards

• Use folders to separate app-level, infra-level, and cluster-level dashboards.

• Tag your dashboards (e.g., k8s, database, alerts) for better searchability.

• Set appropriate permissions if working in a team.

Why Dashboards Matter in DevOps

• Help detect system anomalies before they become incidents.

• Provide real-time operational visibility.

• Improve collaboration between dev, ops, and SRE teams.

• Essential for SLO/SLA compliance tracking.

Final Thoughts

By turning Prometheus metrics into clear and actionable visuals, Grafana becomes an essential tool for proactive system health monitoring and troubleshooting.

Tomorrow, we’ll dive into Alertmanager, the bridge that turns alerts into real-time notifications, ensuring that no anomaly goes unnoticed.

Stay tuned and keep building!