Monitoring your systems isn’t just for big companies—anyone running applications or infrastructure can benefit from real-time observability. In this guide, you'll learn how to set up a full monitoring stack using Prometheus, Alertmanager, and Grafana on a local virtual machine (VM).

🌍 Why Monitoring Matters: A Real-World Example

Imagine you're running a small Node.js API used by a frontend application. Suddenly, users start complaining that it's “slow” or “not working.” With no monitoring, you'd be blind. But with Prometheus and Grafana, you could see CPU spikes, memory usage, and HTTP errors in real-time—giving you instant insight and faster resolution.

🛠️ Step 1: Installing and Configuring Prometheus

1.1 Create the Prometheus User and Directories

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

1.2 Download and Install Prometheus

cd /tmp/
wget https://github.com/prometheus/prometheus/releases/download/v2.7.1/prometheus-2.7.1.linux-amd64.tar.gz
tar -xvf prometheus-2.7.1.linux-amd64.tar.gz
cd prometheus-2.7.1.linux-amd64/

sudo mv console* prometheus.yml /etc/prometheus
sudo mv prometheus promtool /usr/local/bin/
sudo chown -R prometheus:prometheus /etc/prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

1.3 Create a Prometheus systemd Service

sudo vim /etc/systemd/system/prometheus.service

Paste this:

iniCopyEdit[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

🚨 Step 2: Setting Up Alertmanager

2.1 Create User and Directories

sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir /etc/alertmanager

2.2 Download and Install

cd /tmp/
wget https://github.com/prometheus/alertmanager/releases/download/v0.16.1/alertmanager-0.16.1.linux-amd64.tar.gz
tar -xvf alertmanager-0.16.1.linux-amd64.tar.gz
cd alertmanager-0.16.1.linux-amd64/

sudo mv alertmanager amtool /usr/local/bin/
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager /usr/local/bin/amtool
sudo mv alertmanager.yml /etc/alertmanager/
sudo chown -R alertmanager:alertmanager /etc/alertmanager

2.3 Create Alertmanager systemd Service

sudo vim /etc/systemd/system/alertmanager.service

Paste:

iniCopyEdit[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/etc/alertmanager/
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager

2.4 Connect Alertmanager to Prometheus

Edit Prometheus config:

sudo vim /etc/prometheus/prometheus.yml

Update the alerting section:

yamlCopyEditalerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

Then restart Prometheus:

sudo systemctl restart prometheus
sudo systemctl status prometheus

📈 Step 3: Installing Grafana

3.1 Install Grafana

sudo apt-get install libfontconfig
cd /tmp/
wget https://dl.grafana.com/oss/release/grafana_5.4.3_amd64.deb
sudo dpkg -i grafana_5.4.3_amd64.deb

3.2 Start Grafana

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

3.3 Get the VM's IP Address

Run:

ip a

Look for the IP under your main network adapter, usually something like:

nginxCopyEditinet 192.168.x.x

⚙️ If you're using VirtualBox and can’t access Grafana in your browser, you likely need to change your network settings.

Fix: Change VirtualBox Network Mode to "Bridged Adapter"

Why?
Bridged mode allows your VM to behave like a separate device on your local network. This way, you can access Grafana (and Prometheus) from your host machine using the VM’s IP address.

🔄 How to Change to Bridged Adapter

Shut down your VM if it’s currently running.
Open VirtualBox Manager.
Select your VM from the list on the left.
Click Settings > Network.
Under the Adapter 1 tab:
- Ensure Enable Network Adapter is checked.
- Set Attached to: Bridged Adapter.
- Select your host's network interface (e.g., Wi-Fi or Ethernet) from the Name dropdown.
Click OK to save.
Start your VM again.
Run ip a inside the VM to find its new IP address (e.g., 192.168.x.x).

Now, from your host browser, visit:

http://<your_VM_IP>:3000

to access Grafana, and:

http://<your_VM_IP>:9090

to access Prometheus.

3.4 Access Grafana in Browser

Go to:

http://<your_VM_IP>:3000

Default login:

Username: admin
Password: admin (you’ll be asked to change it)

🔗 Step 4: Add Prometheus and Other Endpoints

Open:

sudo vim /etc/prometheus/prometheus.yml

Append to scrape_configs:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'alertmanager'
    static_configs:
      - targets: ['localhost:9093']

  - job_name: 'grafana'
    static_configs:
      - targets: ['<your_VM_IP>:3000']

Restart Prometheus again:

sudo systemctl restart prometheus

✅ Final Check: Prometheus UI

Open:

http://<your_VM_IP>:9090

Go to Status > Targets to ensure all endpoints are listed and their state is UP.

🧠 Honorable Mentions

🟡 Common Prometheus Failures

If Prometheus fails to start, check for YAML syntax errors in prometheus.yml. Common mistakes include:

Misaligned indentation
Forgetting : or -

Always run:

bashCopyEditsudo systemctl restart prometheus
sudo systemctl status prometheus

🛑 Grafana Data Source: “Bad Gateway” Error?

If Grafana throws a 502 Bad Gateway when adding Prometheus as a data source:

Make sure the URL is http://<your_VM_IP>:9090
Ensure Prometheus is running:
```
  sudo systemctl status prometheus
```
Avoid using localhost or 127.0.0.1 in Grafana if it's running in a VM—use the VM’s actual IP.

🎉 Conclusion

Congrats! You've now got a basic but powerful observability stack running locally. You can:

Scrape metrics from services
Send alerts via Alertmanager
Visualize everything with Grafana

How to Set Up Prometheus Monitoring with Alertmanager and Grafana on a Local VM (Step-by-Step)