Prometheus and Grafana Monitoring web server with Docker on AWS EC2

This lab shows how to create monitoring with Prometheus+Grafana for better monitoring for webapp server on EC2 using docker.

This is the diagram for this lab

1. Create web-app EC2 Server

Create EC2 for web-app server on EC2 using new or created KeyPair. I used ubuntu image with t2.micro in this lab for minimum cost.

Create new security group allowing SSH, HTTP and Port 5000

In here for port 5000 if you use Elastic IP for Webapp server you can use Source type as that Prometheus Server EIP. If you don’t use Elastic IP for webapp server the public IP will change every time the node restarts.

Under Advanced Details, use the user data, so then required packages are installed when server is up. Then Launch Instance

#!/bin/bash

# Update and install Docker + Docker Compose
apt update -y
apt install -y docker.io docker-compose

# Add the 'ubuntu' user to the docker group
usermod -aG docker ubuntu

# Enable and start Docker
systemctl enable docker
systemctl start docker

After logging in to the web-app server using the public IP and keypair. Create webapp directory and go inside that directory

mkdir webapp && cd webapp

Create app.py file for webapp test using below code.

from flask import Flask, Response
app = Flask(__name__)

@app.route("/")
def index():
    return "Web server is running"

@app.route("/metrics")
def metrics():
    return Response('webapp_up 1\n', mimetype='text/plain')

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Create Dockerfile from below. You can take reference from https://docs.docker.com/reference/dockerfile/ for Dockerfile.

FROM python:3.9-slim
WORKDIR /app
COPY app.py .
RUN pip install flask
CMD ["python", "app.py"]

Create docker-compose.yml. You can take reference from https://docs.docker.com/reference/compose-file/ for docker-compose.yml

version: '3'
services:
  web:
    build: .
    ports:
      - "5000:5000"

Run

docker-compose up -d --build

-d means detach mode and --build means build the image new instead of reusing the old image. After that use docker ps to check the running docker.

2. Create Prometheus server On EC2 with docker

Create another EC2 for Prometheus server using the above steps. But this time instead of opening port 5000 open port 9090 for Prometheus.

After connecting to Prometheus server,

mkdir prometheus && cd prometheus

Create prometheus.yml file for config of target server. You can take reference from here https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'web-app'
    static_configs:
      - targets: ['<WEB_EC2_PUBLIC_IP>:5000']

Create docker-compose.yml

version: '3'
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

After that check http://<PROMETHEUS_PUBLIC_IP>:9090/targets

you should see the webserver as targe

3. Create Grafana server On EC2 with docker

Create EC2 as above steps for Grafana server. But this time open port 3000 for Grafana Serve

Login to Grafana Server and create grafana directory

mkdir grafana && cd grafana

Create docker-compose.yml . I want to avoid hardcoding the grafana password in docker-compose file so I will use another method by creating .env and add the line GRAFANA_ADMIN_PASSWORD=admin123 use your desired password. Here I used admin123 for grafana admin password. Use .gitignore to ignore uploading to git for .env file so then your password is not uploaded to git.

version: '3'
services:
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}

Then run the following

docker-compose up -d

You can check http://<GRAFANA_PUBLIC_IP>:3000

if Grafana is up, use
username: admin
password: <your input password>

Go to Connections>Data Sources>Prometheus to show data send from Prometheus in Grafana

Go to Dashboard>New Dashboard>Add Visualization>(Select Data Source as Prometheus). You can use metrics for example up for server up in visualization. Then save dashboard

You can see as below as server uptime.

You can also integrate alerting with PagerDuty, Slack etc. I use PagerDuty to test alert

Go to Alert>New Alert Rule. Use metrics as up or webapp_up and when query is Equal to 0 the alert will trigger. Create the required Alert Folder, Evaluation Grp and most importantly create contact points.

Go to Create contact points and choose integration as PagerDuty. And you have to copy the Integration key from PagerDuty and put it here.

In Pager Duty Go to Services>New Service and create a service with desired name.

Choose the Service created and go to Integrations and Add integration. You can choose Events API V2 or Prometheus. Any of them working. Copy the Integration key from Pager Duty and paste in Grafana Contact Point.

Try stopping the webapp container using docker stop <container ID>. After some time you will receive an email from Pager Duty as below.

After all the testing has been finished don’t forget to delete all three EC2 that you used for this lab to avoid additional cost.