Automated Website & Server Monitoring with Django, Celery, Redis and WebSockets

Husnain AliHusnain Ali
6 min read

Introduction

In today's digital landscape, website and server uptime is critical. Downtime can lead to lost revenue, poor user experience, and damaged reputation. To address this, I built Service Monitoring Pro—a Django-based web application that continuously monitors websites and servers,notifies users (using emails) when downtime occurs, and provides an alert dashboard.

In this article, I’ll walk you through how I built this app using Django, Celery, Redis, and Docker.🚀

Monitoring Mechanism

The system is designed to monitor websites and servers using asynchronous background tasks. It works as follows:

  1. Website Monitoring: The system periodically sends HTTP requests to configured websites and checks for responses.

  2. Server Monitoring: It utilizes ping commands to determine server availability.

  3. Failure Detection: If a website or server fails to respond within a set threshold, an alert is triggered.

  4. Alert System: Email notifications are sent to the user when a service goes down.

  5. Real-time Updates: Using WebSockets, the dashboard reflects live status updates of monitored services.

Notification System

Timely alerts are essential in server and website monitoring. Our system ensures that users receive instant notifications through:

  • Email Alerts: Whenever a service becomes unresponsive, an email is sent with details of the failure.

  • Dashboard Alerts: A real-time status update appears on the dashboard to keep users informed at all times.

Implementation Details

Models for Storing Website Data

Create a model to store the websites and servers that we want to monitor. Each entry will store the URL or server address, the status (whether it's up or down), and the frequency at which we want to check it. Here is the model code:

from django.db import models
from django.contrib.auth.models import User
from django.utils.timezone import now

class Website(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    url = models.URLField(unique=True)
    status = models.CharField(max_length=10, default="UNKNOWN")
    interval = models.IntegerField(choices=INTERVAL_CHOICES, default=60) 
    last_checked = models.DateTimeField(auto_now=True)
    monitoring_type=models.CharField(max_length=50,default="server")

    def __str__(self):
        return self.url

Explanation:

  • url: This field stores the URL or address of the website or server.

  • status: This field holds the current status of the website (e.g., "UP", "DOWN").

  • interval: Defines how frequently the system should check the website (e.g., every 1 minute, every 5 minutes).

  • last_checked: The timestamp of the last time the website was checked.

  • monitoring_type: Whether we are monitoring a website or server.

Celery for Asynchronous Task Execution

To check the website’s or server’s status at regular intervals, we will use Celery, a powerful asynchronous task queue. Celery helps run periodic tasks in the background, ensuring that the system can check the website without blocking the user’s workflow.

Creating a file celery.py in your Django project directory:

import os
from celery import Celery
from celery.schedules import crontab
from datetime import timedelta

# Set Django settings for Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "process_monitoring.settings")

app = Celery("process_monitoring")

# Load configuration from Django settings
app.config_from_object("django.conf:settings", namespace="CELERY")

# Make sure Celery is using Redis
app.conf.broker_url = os.getenv("CELERY_BROKER_URL", "redis://redis:6379/0")

# Discover tasks in installed Django apps
app.autodiscover_tasks()

# Define periodic tasks using Celery Beat
app.conf.beat_schedule = {
    'check-websites-every-30-sec': {
        'task': 'monitoring.tasks.check_30_sec',
        'schedule': timedelta(seconds=30),
    },
    'check-websites-every-1-min': {
        'task': 'monitoring.tasks.check_1_min',
        'schedule': timedelta(minutes=1),
    },
    'check-websites-every-2-min': {
        'task': 'monitoring.tasks.check_2_min',
        'schedule': timedelta(minutes=2),
    },
    'check-websites-every-5-min': {
        'task': 'monitoring.tasks.check_5_min',
        'schedule': timedelta(minutes=5),
    },
}

This code sets up Celery for managing periodic tasks in a Django project, specifically for monitoring website statuses at different intervals. It configures Celery with Redis as the message broker, enabling asynchronous task execution in the background without affecting the main application’s performance. The periodic tasks are defined using Celery Beat, which allows tasks like check_30_sec, check_1_min, check_2_min, and check_5_min to run at intervals of 30 seconds, 1 minute, 2 minutes, and 5 minutes, respectively. This configuration ensures that website monitoring occurs automatically at the specified intervals, providing real-time status updates to users.

Create a periodic task to check the website’s status at the interval defined in the Website model. In tasks.py

import requests
import socket
import subprocess
import logging
from django.utils.timezone import now
from django.core.mail import send_mail
from django.conf import settings
from celery import shared_task
from .models import Website
from alert.models import Alert
import monitoring.utils as utils

logger = logging.getLogger(__name__)



def check_websites(interval):
    websites = Website.objects.filter(interval=interval)
    logger.info(f"Checking {websites.count()} websites for {interval}-second interval")

    for website in websites:
        monitoring_type=website.monitoring_type

        if monitoring_type=="website":
            website_status=utils.check_website_status(website.url)
            if website.status!=website_status:
                utils.send_alert(website,website_status)
            website.status=website_status
            website.last_checked=now()
            website.save()
            continue
        elif monitoring_type=="server":
            hostname = website.url.replace("http://", "").replace("https://", "").split("/")[0]
            new_status=utils.check_server_status(hostname,website.url)
            if website.status != new_status:
                utils.send_alert(website, new_status)
            website.status = new_status
            website.last_checked = now()
            website.save()
            continue
        elif monitoring_type=="both":
            hostname = website.url.replace("http://", "").replace("https://", "").split("/")[0]
            new_status_server=utils.check_server_status(hostname,website.url)
            new_status_website=utils.check_website_status(website.url)
            new_status=""
            if new_status_server=="UP" and new_status_website=="UP":
                new_status="UP"
            else:
                new_status="DOWN"
            if website.status!=new_status:
                utils.send_alert(website,new_status)
            website.status=new_status
            website.last_checked=now()
            website.save()
    return f"Checked {websites.count()} websites for {interval}-second interval"

# 6️⃣ CELERY TASKS (Scheduled Checks)
@shared_task
def check_30_sec():
    return check_websites(30)

@shared_task
def check_1_min():
    return check_websites(60)

@shared_task
def check_2_min():
    return check_websites(120)

@shared_task
def check_5_min():
    return check_websites(300)

This tasks.py file defines periodic Celery tasks for checking the status of websites and servers at different intervals. The check_websites function queries the Website model to fetch websites based on their monitoring interval. It checks the status of each website or server (depending on the monitoring_type) by calling helper functions from utils. If there is a change in status (e.g., from UP to DOWN), it triggers an alert using the send_alert function. The four Celery tasks (check_30_sec, check_1_min, check_2_min, and check_5_min) are scheduled to run at 30-second, 1-minute, 2-minute, and 5-minute intervals, respectively, ensuring continuous monitoring.

The utils.py file provides functions for monitoring website and server health:

import requests
import socket
import subprocess
import logging
from django.utils.timezone import now
from django.core.mail import send_mail
from django.conf import settings
from alert.models import Alert
logger = logging.getLogger(__name__)


class ServerChecking:
    def is_server_reachable(self,hostname):
        try:
            output = subprocess.run(["ping", "-n", "2", hostname], capture_output=True, text=True, timeout=5)
            return "Reply from" in output.stdout
        except Exception as e:
            return False


    # 2️⃣ PORT CHECK FUNCTION (Checks if HTTP/HTTPS are Running)
    def is_port_open(self,hostname, port):
        try:
            with socket.create_connection((hostname, port), timeout=5):
                return True
        except (socket.timeout, socket.error):
            return False

    # 3️⃣ WEBSITE STATUS CHECK FUNCTION
    def check_website_status(self,url):
        try:
            response = requests.get(url, timeout=5)
            return response.status_code not in [502, 503, 504]
        except requests.RequestException:
            return False

def check_website_status(url):
    status=""
    try:
        response = requests.get(url, timeout=5)  # Set timeout to avoid long delays
        if response.status_code>=200 and response.status_code<400:
            status="UP"
            return status
        else:
            status="DOWN"
            return status
    except requests.exceptions.ConnectionError:
        status="DOWN"
        return status
    except requests.exceptions.Timeout:
        status="DOWN"
        return status
    except requests.exceptions.RequestException:
        status="DOWN"
        return status


def check_server_status(hostname,url):
    check=ServerChecking()
    server_up = check.is_server_reachable(hostname)
    port_80_open = check.is_port_open(hostname, 80)
    port_443_open = check.is_port_open(hostname, 443)
    website_up = check.check_website_status(url)
    new_status = "UP" if server_up and website_up and (port_80_open or port_443_open) else "DOWN"
    return new_status

def send_alert(website, status):
    message = f"Your server hosting {website.url} is currently {status}."
    Alert.objects.create(user=website.user, url=website.url, status=status, message=message, is_read=False)

    if status == "DOWN":
        try:
            send_alert_email(website.url,website.user.email)
            logger.info(f"Email alert sent for {website.url}")
        except Exception as e:
            logger.error(f"Failed to send alert email for {website.url}: {e}")

def send_alert_email(website_url,email):
    subject = "⚠️ Website Server Down Alert"
    message = f"The website Server {website_url} is currently DOWN. Please check it immediately!"
    from_email = settings.DEFAULT_FROM_EMAIL
    recipient_list = [email]  # Change this to the user's email

    try:
        send_mail(subject, message, from_email, recipient_list)
        logger.info(f"Alert email sent for {website_url}")
    except Exception as e:
        logger.error(f"Failed to send email for {website_url}: {e}")

  1. ServerChecking Class:

    • is_server_reachable: Pings the server to check if it's reachable.

    • is_port_open: Checks if HTTP (port 80) or HTTPS (port 443) is open.

    • check_website_status: Checks if a website is accessible via a GET request.

  2. check_website_status: A function that checks website accessibility and returns "UP" or "DOWN" based on the status code or errors.

  3. check_server_status: Combines the checks from ServerChecking to assess the server and website status, returning "UP" or "DOWN".

  4. send_alert: Creates an alert and sends an email notification if the website's status is "DOWN".

  5. send_alert_email: Sends an email alert to the user about the website being down.

In essence, the file monitors the health of websites/servers and sends notifications if issues are detected.

Want to check out the code? Here’s the GitHub Repo.

10
Subscribe to my newsletter

Read articles from Husnain Ali directly inside your inbox. Subscribe to the newsletter, and don't miss out.

ServiceMonitoringDjangoceleryRediswebsockets#ServerMonitoringWebsiteMonitoring

Written by

Husnain Ali
Husnain Ali