A guide on setting up Prometheus, Grafana, and Alert Manager as endpoints.

The Metrics Gatherer, Prometheus

What it is:
An open-source monitoring and alerting tool called Prometheus is made to gather time-series data, or metrics, from your infrastructure, services, and apps.

Important attributes:

• Uses HTTP endpoints (/metrics) to retrieve metrics.

• Maintains information in a time-series database.

• Enables Prometheus Query Language (PromQL) querying.

• Able to produce alerts according to specified rules.

Grafana: The Tool for Visualization

What it is: Grafana is a dashboard and visualization application that facilitates the creation of stunning and engaging dashboards using data from Prometheus (or other sources).

Important attributes:

• Supports a variety of data sources, including MySQL, Prometheus, Loki, and others.

• Dashboards that can be customized with tables, graphs, and charts.

• Visualization alerts and annotations.

• Permissions and user management.

The Alert Dispatcher, or Alertmanager

What it is: Alertmanager oversees the routing, grouping, silence, and notification of alerts produced by Prometheus.

Important attributes:

• Gets notifications from Prometheus.

• Notifies users via Slack, PagerDuty, email, and other channels.

• Combines related alerts to prevent spam.

• Encourages inhibition and alert silencing.

Setting up Prometheus, Grafana, and Alertmanager as monitoring endpoints involves several steps. Below is a step-by-step guide assuming you're running everything on a Linux server (like Ubuntu).

Step 1: Install Prometheus

1. Create Prometheus user and directories:

    sudo useradd --no-create-home --shell /bin/false prometheus
    sudo mkdir /etc/prometheus
    sudo mkdir /var/lib/prometheus
    sudo chown prometheus:prometheus /var/lib/prometheus
   

2. Download and extract Prometheus:

    cd /tmp
    wget https://github.com/prometheus/prometheus/releases/download/v2.7.1/prometheus-2.7.1.linux-amd64.tar.gz
    tar -xvf prometheus-2.7.1.linux-amd64.tar.gz
    cd prometheus-2.7.1.linux-amd64/
   

3. Move the configuration file and set the owner to the Prometheus user:

    cd prometheus-2.7.1.linux-amd64/
    sudo mv console* /etc/prometheus
    sudo mv prometheus.yml /etc/prometheus
    sudo chown -R prometheus:prometheus /etc/prometheus
   

4. Move binaries and config:

    sudo mv prometheus /usr/local/bin/
    sudo mv promtool /usr/local/bin/
    sudo chown prometheus:prometheus /usr/local/bin/prometheus
    sudo chown prometheus:prometheus /usr/local/bin/promtool
   

5. Create a systemd service file:

    sudo vim /etc/systemd/system/prometheus.service
    type in the file:
    [Unit]
    Description=Prometheus
    Wants=network-online.target
    After=network-online.target
   
    [Service]
    User=prometheus
    Group=prometheus
    Type=simple
    ExecStart=/usr/local/bin/prometheus \
        --config.file /etc/prometheus/prometheus.yml \
        --storage.tsdb.path /var/lib/prometheus/ \
        --web.console.templates=/etc/prometheus/consoles \
        --web.console.libraries=/etc/prometheus/console_libraries
   
    [Install]
    WantedBy=multi-user.target
   

   Save and exit the file by pressing Escape followed by :wq

6. Start Prometheus:

    sudo systemctl start prometheus
    sudo systemctl enable prometheus
    sudo systemctl status prometheus
   

7. To access the Prometheus GUI, search on your browser for http://localhost:9090.

Step 2: Install Alert Manager

1. Download and install:

    sudo useradd --no-create-home --shell /bin/false alertmanager
    sudo mkdir /etc/alertmanager
    cd /tmp/
    wget "https://github.com/prometheus/alertmanager/releases/download/v0.16.1/alertmanager-0.16.1.linux-amd64.tar.gz"
    tar -xvf alertmanager-0.16.1.linux-amd64.tar.gz
    cd alertmanager-0.16.1.linux-amd64/
   

2. Move binaries and set ownership:

    sudo mv alertmanager /usr/local/bin/
    sudo mv amtool /usr/local/bin/
    sudo chown -R alertmanager:alertmanager /usr/local/bin/alertmanager
    sudo chown -R alertmanager:alertmanager /usr/local/bin/amtool
    sudo mv alertmanager.yml /etc/alertmanager/
    sudo chown -R alertmanager:alertmanager /etc/alertmanager/
   

3. Create a systemd service file:

    sudo vim /etc/systemd/system/alertmanager.service
    type in the file:
    [Unit]
    Description=Alertmanager
    Wants=network-online.target
    After=network-online.target
   
    [Service]
    User=alertmanager
    Group=alertmanager
    Type=simple
    WorkingDirectory=/etc/alertmanager/
    ExecStart=/usr/local/bin/alertmanager \
        --config.file=/etc/alertmanager/alertmanager.yml
   
    [Install]
    WantedBy=multi-user.target
   

   Save and exit the file by pressing Escape followed by :wq

4. Start Alertmanager:

   Stop Prometheus, change back to your main directory, and then update the Prometheus configuration file to use Alertmanager using

    sudo systemctl stop prometheus
    cd
    sudo vim /etc/prometheus/prometheus.yml
    Un-comment alertmanager, 
    and change alertmanager to localhost (so it will look like the following):
    alerting:
      alertmanagers:
        - static_configs:
            - targets:
              - localhost:9093
   

   Save and exit the file by pressing Escape followed by :wq

   Reload systemd, and then start the prometheus and alertmanager services

    sudo systemctl daemon-reload
    sudo systemctl start prometheus
    sudo systemctl start alertmanager
    sudo systemctl enable alertmanager
   

Step 3: Install Grafana

1. Add Grafana repo and install:

    sudo apt-get install libfontconfig
    cd /tmp/
    wget https://dl.grafana.com/oss/release/grafana_5.4.3_amd64.deb
    sudo dpkg -i grafana_5.4.3_amd64.deb
   

2. Start Grafana:

    sudo systemctl start grafana-server
    sudo systemctl enable grafana-server
   

3. Access Grafana:

   • URL: http://localhost:3000

   • Default login: admin/admin

Step 4: Adding data sources and configuring endpoints.

1. Click Add data source on the Grafana home page.

2. Select Prometheus.

3. In the HTTP section, set the URL to http://localhost:9090

   Click Save & Test.

4. Back in the monitoring server terminal, open the Prometheus configuration file:

    sudo vim /etc/prometheus/prometheus.yml
   

   At the end of the file, at the bottom of the scrape_configs section, add the Alertmanager endpoint (make sure it aligns with the - job_name: 'prometheus' line):

    - job_name: 'alertmanager'
      static_configs:
      - targets: ['localhost:9093']
   

   Beneath what you just added, add the Grafana endpoint (using the public IP address of the Grafana server):

    - job_name: 'grafana'
      static_configs:
      - targets: ['localhost:3000']
   

   The whole code should be aligned this way:

    # my global config
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
   
    # Alertmanager configuration
    alerting:
      alertmanagers:
        - static_configs:
            - targets:
                - localhost:9093
   
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"
   
    # Scrape configs
    scrape_configs:
      - job_name: "prometheus"
        static_configs:
          - targets: ["localhost:9090"]
   
      - job_name: "alertmanager"
        static_configs:
          - targets: ["localhost:9093"]
   
      - job_name: "grafana"
        static_configs:
          - targets: ["localhost:3000"]
   

   Save and exit the file by pressing Escape followed by :wq.

   Restart Prometheus:

    sudo systemctl restart prometheus
    sudo systemctl status prometheus
   

   Using the public IP address of the monitoring server, navigate to the Prometheus web UI in a new browser tab: http://localhost:9090.

   Click Status > Targets.

   Ensure all three endpoints are listed on the Targets page.