Intro

Goal

I have an external HDD, and I want to have a copy of it in the cloud.

I do not want

I do not want to connect the external HDD to the laptop, run some commands to back up, monitor the status, wait until it's done, etc.

I want

• Connect external HDD to Raspberry Pi

• Receive notification on Telegram that backup has started

• Receive notification on Telegram that backup is done

• Safely disconnect HDD

How would I achieve this?

• Configure Telegram bot to receive notifications.

• Configure Raspberry Pi so it has a connection to the remote location where we are going to store the backup online (in my case, it will be Hetzner Storage Box).

• Configure Raspberry Pi so it automatically mounts the external HDD when it is connected.

• Configure Raspberry Pi so it automatically starts the backup script when the external HDD is connected.

• Configure a Python script that will run the backup and send notifications to Telegram when the backup starts and finishes.

• Configure Raspberry Pi so it automatically unmount the external HDD when it is not in use.

Disclaimer

• The flow can always be better, but it is good enough for me.

• The rules or script can be better, but it is good enough for me.

• The described article might miss some steps; this article is more documentation for myself.

Configure Telegram API token

To receive messages on Telegram, we need to create a Telegram bot.

This can be done using @BotFather. Create a new bot and save the token.

Then, create a new private group and add the newly created Telegram bot to it.

Save the ChatID of this group; you can find it in the group info.

To see ChatID got to Settings -> Advanced -> Experimental settings -> Show Peer ID in Profile

Configure rclone

For backup, we can use many methods; for now, I choose rclone.

Install it

sudo apt install rclone

Configure it

rclone config

In my case, I configured SFTP to Hetzner Storage Box. Later, we will use the rclone sync command. The good thing is that it does not care which backup you use, so rclone is a kind of universal solution.

Auto mount with fstab and systemd

By default, when you connect an external HDD, you need to mount it. You can create a record in Fstab to automatically mount the disk after the system starts, but that's not really what we want. We want the disk to mount automatically just after connecting the external HDD to USB.

Based on the article Automount filesystems with systemd, we are going to set up Fstab so it automatically mounts the external HDD when it is connected and automatically unmounts when the disk is not used for more than 60 seconds. This is perfect for my use case.

Create a directory where the external HDD will be mounted.

mkdir -p /mnt/external-hdd

Get UUID of external disk

blkid /dev/sda1

Note UUID and TYPE , we will use it later.

Make backup of /etc/fstab

sudo cp /etc/fstab /etc/fstab.backup

Add entry to /etc/fstab

UUID=5467CF5C215FB31F /mnt/external-hdd ntfs defaults,x-systemd.automount,x-systemd.idle-timeout=60s 0 2

Explanation:

• UUID: from blkid

• ntfs: is TYPE from blkid

• x-systemd.automount: to delegate this mount to systemd.

• x-systemd.idle-timeout=60s: configures an idle timeout. Once the mount has been idle for the specified time, systemd will attempt to unmount. So after 60 seconds of non-use of the mount point, it will be umounted. Understand that a simple cd command in a shell on that mountpoint will avoid the idle timeout, even if no apparent activity occurs. See x-systemd.idle-timeout in manpage for details.

• 0: the filesystem will not be included in dump.

• 2: the filesystem will be checked by fsck after filesystems with a priority of 1.

Start systemd unit

Yes, there is no need to create a systemd unit file. The unit is created automatically, and it has a name like in your fstab entry, in my case /mnt/external-hdd.

sudo systemctl start mnt-external\\x2dhdd.automount 
sudo systemctl status mnt-external\\x2dhdd.automount

Disconnect and reconnect the external HDD, test if it works, and you should see your files.

ls -l /mnt/external-hdd

Configure running backup script automatically

Udev part

Now we need to set up automation that will start the Python script as soon as the external HDD is connected. For this, we will create a udev rule.

Create udev file

sudo vim /etc/udev/rules.d/99-auto-backup.rules

Content of the file

ACTION=="add", KERNEL=="sd[a-z][1-9]", ENV{ID_FS_UUID}=="5467CF5C215FB31F", RUN+="/bin/systemctl start auto-backup-external-hdd.service"

Apply udev rule

sudo udevadm control --reload-rules sudo udevadm trigger

Systemd unit for running backup script

This systemd unit will be started by the udev rule from above.

Create systemd unit.

sudo vim /etc/systemd/system/auto-backup-external-hdd.service

Content of file

[Unit]
Description=Auto Backup External HDD
After=multi-user.target

[Service]
User=piuser
ExecStart=/usr/bin/python3 /home/piuser/auto-backup-external-hdd/auto_backup_external_hdd.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

Note ExecStart:

• /usr/bin/python3 is the path for Python, check it with which python3

• path to the Python script which will run automation (we will create it below)

Python script to run backup

Config file for script

Create the ~/auto-backup-external-hdd/auto_backup_config.ini file where we will store configs for the script.

Example of the file:

[Telegram]
Token = 123:xyz
ChatID = -456

[RemoteStorage]
Host = hetzner-storage-box-u987

Note that ChatID should have a - sign.

See above how to get the Telegram token and chatid.

Python script

Install python lib for telegram

pip install python-telegram-bot --upgrade --pre --break-system-packages

I install it via pip and not apt because apt does not have the latest version of the lib, so --break-system-packages is used.

I use --pre to have the latest (more than v20) version of python-telegram-bot.

Create script

Path for the script auto-backup-external-hdd/auto_backup_external_hdd.py

"""
Script will do the following:
- send notification to Telegram that backup process started
- run rclone to start syncing external HDD to remote storage
- send notification to Telegram that backup is completed

Usage:
- Create auto_backup_config.ini with content, example:
[Telegram]
Token = 123:xyz
ChatID = -456
[RemoteStorage]
Host = hetzner-storage-box-u987
"""
import os
import subprocess
import logging
import configparser
import asyncio
from typing import Tuple
from telegram import Bot

# Configure logging to stdout
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s",
    handlers=[logging.StreamHandler()],
)

# Suppress logs from 'telegram' and 'httpx'
logging.getLogger("telegram").setLevel(logging.WARNING)
logging.getLogger("httpx").setLevel(logging.WARNING)

# Constants for backup
SOURCE_DIR: str = "/mnt/external-hdd/"
DESTINATION_DIR: str = "/external-hdd-1"
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
CONFIG_FILE = os.path.join(SCRIPT_DIR, "auto_backup_config.ini")

def read_telegram_config(config_file: str) -> Tuple[str, str]:
    """
    Read Telegram token and chat ID from a configuration file.

    Args:
        config_file (str): Path to the configuration file.

    Returns:
        Tuple[str, str]: Telegram token and chat ID.

    Raises:
        KeyError: If required configuration keys are missing.
    """
    config = configparser.ConfigParser()
    config.read(config_file)
    try:
        # Read Telegram configuration
        token = config["Telegram"]["Token"]
        chat_id = config["Telegram"]["ChatID"]
        # Read RemoteStorage configuration
        remote_host = config["RemoteStorage"]["Host"]
        return token, chat_id, remote_host
    except KeyError as e:
        logging.error("Missing configuration key: %s", e)
        raise


async def send_message(bot: Bot, chat_id: str, message: str) -> None:
    """
    Send a message to a Telegram chat.

    Args:
        bot (Bot): Telegram Bot instance.
        chat_id (str): ID of the chat to send the message to.
        message (str): Message to send.

    Raises:
        Exception: If sending the message fails.
    """
    try:
        await bot.send_message(chat_id=chat_id, text=message)
        logging.info("Telegram message sent: %s", message)
    except Exception as e:
        logging.error("Failed to send Telegram message: %s", e)


async def main() -> None:
    """
    Main function to handle the backup process and Telegram notifications.
    """
    # Read Telegram credentials
    try:
        telegram_token, chat_id, remote_host = read_telegram_config(CONFIG_FILE)
        bot = Bot(token=telegram_token)
    except Exception as e:
        logging.error("Failed to initialize Telegram Bot: %s", e)
        return

    # Check if SOURCE_DIR is empty
    if not os.listdir(SOURCE_DIR):
        empty_message = f"⚠️ SOURCE_DIR {SOURCE_DIR} is empty. Backup process aborted."
        logging.warning(empty_message)
        await send_message(bot, chat_id, empty_message)
        return

    destination = f"{remote_host}:{DESTINATION_DIR}"
    logging.info("Backup process started for %s to %s", SOURCE_DIR, destination)
    await send_message(bot, chat_id, f"📂 Backup started from {SOURCE_DIR} to {destination}")

    try:
        # Run rclone sync
        rclone_command = [
            "rclone",
            "sync",
            SOURCE_DIR,
            destination,
            "--progress",
        ]
        logging.info("Running command: %s", " ".join(rclone_command))
        subprocess.run(rclone_command, check=True)

        # Notify completion
        success_message = "✅ Backup to Hetzner Storage Box completed successfully!"
        logging.info(success_message)
        await send_message(bot, chat_id, success_message)
    except subprocess.CalledProcessError as e:
        error_message = f"❌ Backup failed! Error: {e}"
        logging.error(error_message)
        await send_message(bot, chat_id, error_message)
    except Exception as ex:
        error_message = f"⚠️ Unexpected error: {ex}"
        logging.error(error_message)
        await send_message(bot, chat_id, error_message)
    finally:
        logging.info("Backup process finished.")


if __name__ == "__main__":
    asyncio.run(main())

Test it and debug it

Connect the external HDD. You should receive a Telegram message that the backup has started.

You can use these commands for debugging:

sudo systemctl status auto-backup-external-hdd.service
ls -l /mnt/external-hdd
sudo systemctl status mnt-external\\x2dhdd.automount
journalctl -u udev
du -sh /mnt/external-hdd
df -h

Result

We connect the external HDD to the Raspberry Pi and:

• The mnt-external\\x2dhdd.automount systemd unit automatically mounts the disk to /mnt/external-hdd.

• The 99-auto-backup.rules udev rule starts the auto-backup-external-hdd.service systemd unit.

• The auto-backup-external-hdd.service systemd unit starts the auto_backup_external_hdd.py Python script.

• The auto_backup_external_hdd.py Python script sends a message to Telegram that the backup has started, then it runs rclone to sync the disk to remote storage, and we receive a message on Telegram that the backup is done.

• The disk automatically unmounts after 60 seconds and can be safely disconnected.

In a shorter way, without technical stuff

• Connect the disk.

• Receive a message that the backup is done.

• Disconnect the disk 🎉