Setting Up a Monitoring Stack in Docker Compose

This guide provides instructions on setting up a comprehensive monitoring stack using Grafana, Prometheus, Node Exporter, cAdvisor and Loki. These components are orchestrated with Docker Compose and exposed via an NGINX reverse proxy, making them accessible through a single domain.

Components

• Grafana: The analytics and monitoring solution with support for multiple data sources, including Prometheus.

• Prometheus: The monitoring and alerting toolkit, collecting metrics from configured targets at specified intervals.

• Node Exporter: A Prometheus exporter for hardware and OS metrics exposed by *NIX kernels.

• cAdvisor: Analyzes resource usage and performance characteristics of running containers.

• Loki: A horizontally-scalable, highly-available, multi-tenant log aggregation system.

• NGINX: Used as a reverse proxy to expose Grafana on the internet securely.

Prerequisites

• Docker and Docker Compose installed on your host machine.

• Domain name configured to point to the host machine (for NGINX configuration).

Configuration Files Overview

• docker-compose.yml: Defines the services, networks, and volumes for the Docker containers.

• prometheus.yml: Configuration file for Prometheus to define scrape targets and intervals.

• grafana.ini: Configuration file for Grafana's settings, including SMTP for email notifications.

• loki-config.yml: Configuration file for Loki to define the storage backend and other settings.

• domain.conf: NGINX configuration for proxying requests to Grafana and handling WebSocket connections.

Installation Steps

1. Configure Docker Daemon: Enable Docker metrics by adding the following to /etc/docker/daemon.json and restarting Docker:

    {
      "metrics-addr": "127.0.0.1:9323"
    }
    # Then reload Docker configurations:
    sudo systemctl reload docker
   

2. Install Loki plugin for Docker: Install the Loki plugin for Docker to enable log collection:

    docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
   

3. Clone the Repository: Clone the repository to your host machine:

    git clone https://gist.github.com/398e94fb0aebb994b25223d99e7ae769.git monitoring-stack
    cd monitoring-stack
   

4. Start the Services: Start the monitoring stack using Docker Compose:

    docker-compose up -d
   

5. Access Grafana: Access Grafana at http://<your-domain> and log in with the default credentials (username: admin, password: admin).

Important Notes:

• Replace placeholder values in grafana.ini with your SMTP credentials to enable email notifications.

• Adjust the domain.conf to match your domain and specific requirements.

• Review Docker and NGINX logs in case of any issues during setup.

6. How to send container logs to loki:

    docker run --name nginx-loki -d \
    -p 8080:80 \
    --log-driver=loki \
    --log-opt loki-url=http://localhost:3100/loki/api/v1/push \
    nginx:latest