What is Grafana?

Grafana is an open-source tool for monitoring and visualizing data from various sources, such as Prometheus, InfluxDB, and MySQL. It lets you create interactive dashboards to track system metrics, logs, and real-time performance. It is commonly used in DevOps for infrastructure and application monitoring.

Common Use Cases:

• Infrastructure Monitoring

• Application Performance Monitoring (APM)

• Business Metrics Dashboards

• IoT Data Visualization

What are the features of Grafana?

Here are the key features of Grafana:

1. Custom Dashboards – Create interactive and visually appealing dashboards with charts, graphs, and tables.

2. Multiple Data Sources – Supports Prometheus, InfluxDB, Elasticsearch, MySQL, PostgreSQL, and more.

3. Alerting – Set up alerts to get notified via email, Slack, or other tools when data crosses defined thresholds.

4. Templating – Use variables to build reusable and dynamic dashboards.

5. Plugins – Extend functionality with plugins for panels, data sources, and apps.

6. User Management – Role-based access control (Admin, Editor, Viewer) for team collaboration.

7. Annotations – Add notes to graphs for events or incidents to give context to your data.

8. Snapshot Sharing – Share dashboard snapshots with others securely.

9. Cross-platform – Works on Linux, Windows, and macOS.

10. Open-source & Extensible – Free to use and customizable for various use cases.

Why Grafana?

Grafana is used because it makes monitoring easy, visual, and powerful. It helps you:

1. Understand data quickly with real-time, interactive dashboards.

2. Track system health and performance across multiple sources.

3. Get alerted when something goes wrong (e.g., server goes down).

4. Collaborate with your team through shared dashboards and role-based access.

5. Support multiple tools in one place (like Prometheus, MySQL, Elasticsearch, etc.).

What type of monitoring can be done via Grafana?

Grafana supports a wide range of monitoring types, including:

1. Infrastructure Monitoring

   • Track servers, virtual machines, CPU, memory, disk, and network usage.

2. Application Monitoring

   • Monitor application performance, request rates, error rates, and response times.

3. Container & Orchestration Monitoring

   • Monitor Docker containers and Kubernetes clusters (pods, nodes, deployments, etc.).

4. Database Monitoring

   • Track database performance metrics like queries, connections, read/write speeds, etc.

5. Network Monitoring

   • Visualize network traffic, latency, and packet loss.

6. Cloud Services Monitoring

   • Monitor AWS, Azure, and GCP services using cloud-specific exporters or integrations.

7. Log Monitoring

   • Use tools like Loki with Grafana to collect, search, and analyze logs.

8. IoT Monitoring

   • Visualize sensor data and monitor smart devices in real time.

9. Business Metrics Monitoring

   • Track KPIs like user sign-ups, revenue, conversions, etc.

What databases work with Grafana?

Grafana supports a wide range of databases and data sources, including:

Time-Series Databases (commonly used for monitoring):

• Prometheus

• InfluxDB

• Graphite

• OpenTSDB

• VictoriaMetrics

SQL Databases:

• MySQL

• PostgreSQL

• Microsoft SQL Server (MSSQL)

• SQLite

NoSQL Databases:

• MongoDB (via plugins)

• Cassandra (limited support via plugins)

Cloud & Monitoring Services:

• Amazon CloudWatch

• Google Cloud Monitoring

• Azure Monitor

• New Relic

• Datadog

Logging & Tracing Tools:

• Loki (for logs)

• Tempo (for traces)

• Elasticsearch (for logs and metrics)

What are metrics and visualizations in Grafana?

---

Metrics in Grafana:

• Metrics are the raw data points you collect from systems, applications, databases, etc.

• Examples:

  • CPU usage

  • Memory consumption

  • Network traffic

  • Error rate

  • Request duration

Grafana queries these metrics from data sources like Prometheus, InfluxDB, MySQL, etc., using specific query languages (like PromQL for Prometheus).

---

Visualizations in Grafana:

• Visualizations are the graphical representations of your metrics.

• Types of visualizations:

  • Graphs/Line Charts – For trends over time

  • Bar Charts – For comparing categories

  • Gauge – For showing a value against a range

  • Stat Panel – For single-value metrics (e.g., current CPU usage)

  • Table – For tabular data

  • Heatmaps, Pie charts, and Logs view – For more specialized needs

Visualizations help you analyze trends, spot anomalies, and make decisions quickly.

---

What is the difference between Grafana vs Prometheus?

---

🔹 Grafana

• Type: Visualization & Dashboard Tool

• Purpose: Display and analyze metrics through graphs, charts, and dashboards.

• Data Storage: Does not store data; it queries data from sources like Prometheus, InfluxDB, etc.

• Main Use: Visualizing data, setting up dashboards, and creating alerts.

• Example: You use Grafana to create a dashboard that shows CPU usage trends from Prometheus data.

---

🔹 Prometheus

• Type: Monitoring & Time-Series Database

• Purpose: Used to collect, store, and query metrics from systems and services.

• Data Storage: Stores time-series data locally using its database.

• Main Use: Scraping metrics from targets (like servers, apps), storing them, and enabling alerts.

• Example: Prometheus scrapes metrics like memory usage every 15 seconds from a server.

---

Set up Grafana in your local environment on AWS EC2:

---

✅ Step 1: Launch an EC2 Instance

1. Go to AWS Console → EC2 → Launch Instance.

2. Choose an OS:

   • Amazon Linux 2 (or) *Ubuntu (20.04/22.04)

3. Choose t2.micro (Free Tier).

4. Configure:

   • Allow HTTP, HTTPS, and port 3000 in Security Groups.

5. Launch the instance and SSH into it.

---

✅ Step 2: Update the System

For Amazon Linux 2:

sudo yum update -y

For Ubuntu:

sudo apt update && sudo apt upgrade -y

---

✅ Step 3: Install Grafana

For Amazon Linux 2:

sudo tee /etc/yum.repos.d/grafana.repo<<EOF
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
EOF

sudo yum install grafana -y

For Ubuntu:

sudo apt install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt update
sudo apt install grafana -y

---

✅ Step 4: Start Grafana

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

---

✅ Step 5: Access Grafana

1. Open browser → http://<your-ec2-public-ip>:3000

2. Default login:

   • Username: admin

   • Password: admin

3. Change the password when prompted.

---

✅ (Optional) Open Port 3000 in Security Group

If not done earlier:

• Go to EC2 → Security Groups → Inbound Rules → Add Rule:

  • Type: Custom TCP

  • Port Range: 3000

  • Source: 0.0.0.0/0 (or restrict as needed)

---

Connecting EC2 with Grafana

Connect a Linux and one Windows EC2 instance with Grafana and monitor the different components of the server:

You can connect a Linux and a Windows EC2 instance to Grafana and monitor their components like CPU, memory, and disk usage using Prometheus as the data source:

---

Objective

Monitor:

• Linux EC2 using Node Exporter

• Windows EC2 using Windows Exporter

• Prometheus to collect metrics

• Grafana to visualize the data

---

Requirements

• One Linux EC2 instance for Prometheus + Grafana

• One Linux EC2 instance to monitor

• One Windows EC2 instance to monitor

• Open these ports in respective Security Groups:

  • Grafana: 3000

  • Prometheus: 9090

  • Node Exporter: 9100

  • Windows Exporter: 9182

---

Step-by-Step Setup

1. Set Up Prometheus and Grafana on Linux EC2

Install Prometheus:

wget https://github.com/prometheus/prometheus/releases/latest/download/prometheus-2.51.0.linux-amd64.tar.gz
tar -xvf prometheus-*.tar.gz
cd prometheus-*
sudo cp prometheus /usr/local/bin/
sudo cp promtool /usr/local/bin/
sudo mkdir /etc/prometheus
sudo cp -r consoles/ console_libraries/ /etc/prometheus/
sudo cp prometheus.yml /etc/prometheus/

Edit prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'linux-node'
    static_configs:
      - targets: ['<Linux-EC2-Private-IP>:9100']

  - job_name: 'windows-node'
    static_configs:
      - targets: ['<Windows-EC2-Private-IP>:9182']

Start Prometheus:

./prometheus --config.file=/etc/prometheus/prometheus.yml

Install Grafana:

sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt update
sudo apt install grafana -y
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Access Grafana at http://<public-ip>:3000 with default credentials admin / admin.

---

2. Set Up Linux EC2 for Monitoring

Install Node Exporter:

wget https://github.com/prometheus/node_exporter/releases/latest/download/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xvf node_exporter-*.tar.gz
cd node_exporter-*
./node_exporter &

Node Exporter runs on port 9100.

---

3. Set Up Windows EC2 for Monitoring

Install Windows Exporter:

• Download the latest .msi Installer from GitHub - windows_exporter

• Run the installer and allow it to install as a service.

• By default, it listens on port 9182.

Allow port 9182 in the Windows Firewall and EC2 Security Group.

---

4. Connect Prometheus to Grafana

• Open Grafana

• Navigate to Settings > Data Sources > Add Prometheus

• URL: http://localhost:9090 (if Prometheus is on the same server)

• Click Save and Test

---

5. Import Dashboards

Go to Dashboards > Import

Use the following IDs to import:

• Linux Node Exporter Dashboard: 1860

• Windows Exporter Dashboard: 2129 or 14678

You can now monitor CPU, memory, disk usage, uptime, network traffic, and more for both Linux and Windows EC2 instances through Grafana.

Sending Docker Logs to Grafana:

Task:

• Install Docker and start docker service on a Linux EC2 through USER DATA.

• Create 2 Docker containers and run any basic application on those containers (A simple to-do app will work).

• Now, integrate the docker containers and share the real-time logs with Grafana (Your Instance should be connected to Grafana, and the Docker plugin should be enabled on Grafana).

• Check the logs or docker container name on Grafana UI.

Here’s a step-by-step approach to deploying Docker containers on a Linux EC2 instance using user data, running a simple app, and integrating Docker logs with Grafana for real-time monitoring.

---

TASK BREAKDOWN

1. Install Docker on Linux EC2 (via user data)

2. Run 2 Docker containers with a simple to-do app (or similar)

3. Install and run Promtail (to collect Docker logs)

4. Connect Promtail to Grafana (with Loki)

5. Enable the Docker plugin in Grafana

6. View Docker logs in Grafana UI

---

STEP 1: Launch EC2 with User Data

Use the following User Data script (Amazon Linux 2):

#!/bin/bash
# Update and install Docker
yum update -y
amazon-linux-extras install docker -y
systemctl start docker
systemctl enable docker
usermod -aG docker ec2-user

# Run two simple to-do app containers
docker run -d --name todo-app-1 -p 3001:3000 ghcr.io/docker-samples/educational-todo-app
docker run -d --name todo-app-2 -p 3002:3000 ghcr.io/docker-samples/educational-todo-app

# Install Promtail (log forwarder to Loki)
cd /opt
curl -LO https://github.com/grafana/loki/releases/latest/download/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 promtail
chmod +x promtail

# Create config file for Promtail
cat > /opt/promtail-config.yml <<EOL
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://<GRAFANA-EC2-PRIVATE-IP>:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: docker
          __path__: /var/lib/docker/containers/*/*.log
EOL

# Run Promtail
/opt/promtail -config.file=/opt/promtail-config.yml &

Replace <GRAFANA-EC2-PRIVATE-IP> with your actual private IP where Grafana + Loki is hosted.

---

✅ STEP 2: Install Loki and Enable Docker Logs in Grafana

On the Grafana server:

1. Install Loki:

docker run -d --name=loki -p 3100:3100 \
  -v /etc/loki:/etc/loki \
  grafana/loki:2.9.0 \
  -config.file=/etc/loki/local-config.yaml

2. Add Loki as a Data Source in Grafana:

   • Go to Grafana UI > Settings > Data Sources

   • Add Loki

   • URL: http://localhost:3100

   • Save & Test

3. Install Docker Data Source Plugin (if needed):

   • In Grafana, go to Plugins, search for Docker, and install it.

---

✅ STEP 3: Check Real-Time Logs in Grafana

1. Go to Grafana > Explore

2. Select the Loki data source

3. In the query field, try:

    {job="docker"}
   

4. You should see logs coming from your Docker containers (e.g., todo-app-1, todo-app-2)

5. You can filter by container name or create dashboards showing logs over time

---