⚙️ Day 8 of #150DaysOfDevOps – Service Management with systemd (SYSTEMCTL & JOURNALCTL)

Welcome to Day 8 of the #150DaysOfDevOps challenge. Today, we dive deep into systemd—the powerful init system and service manager used in almost all modern Linux distributions like Ubuntu, CentOS, RHEL, Fedora, and Debian.

---

🧠 Why Learn systemd?

In a DevOps or Linux engineering role, understanding systemd is crucial because it handles:

• Service lifecycle management (start, stop, restart)

• Boot-time service orchestration

• Logging and diagnostics

• Custom service and job definitions

• System performance and uptime optimization

By mastering systemd, you ensure reliable automation, smooth deployments, and simplified troubleshooting in production environments.

---

🧱 What is a systemd Unit?

A unit is any resource that systemd knows how to manage: services, timers, devices, sockets, mounts, etc. Units are defined in unit files.

The most common type is a .service file, which describes how a background process or application should run.

Location of Unit Files:

• /etc/systemd/system/ → Custom or user-defined services

• /lib/systemd/system/ → Installed by packages

• /run/systemd/system/ → Runtime-generated

Use systemctl cat <unit> to inspect a unit’s full definition.

---

🔧 Anatomy of a .service File

📁 Example: /etc/systemd/system/project-mercury.service

[Unit]
Description=Python Django for Project Mercury
Documentation=http://wiki.caleston-dev.ca/mercury
After=postgresql.service

[Service]
ExecStart=/usr/bin/project-mercury.sh
User=project_mercury
Restart=on-failure
RestartSec=10

[Install]
WantedBy=graphical.target

✅ Section Breakdown:

[Unit]

• Description: Appears in status outputs for identification

• Documentation: Links to help or internal wiki

• After: Ensures PostgreSQL starts before this service

[Service]

• ExecStart: Actual command used to run the application

• User: Specifies under which user account the service runs (improves security and resource isolation)

• Restart: Defines restart policy (e.g., on-failure, always, no)

• RestartSec: Delay (in seconds) before attempting restart

• (Optional) ExecStop: Command to run on service termination

[Install]

• WantedBy: Target under which the service should be enabled (e.g., multi-user.target for headless servers)

---

🔍 Creating a Custom systemd Service

Let’s say you have a script backup.sh that runs a database backup.

Step 1: Place your script

sudo mv backup.sh /usr/local/bin/backup.sh
sudo chmod +x /usr/local/bin/backup.sh

Step 2: Create the service file

sudo nano /etc/systemd/system/db-backup.service

[Unit]
Description=Nightly DB Backup
After=network.target

[Service]
ExecStart=/usr/local/bin/backup.sh
User=backupuser
Restart=on-failure

[Install]
WantedBy=multi-user.target

Step 3: Enable and run

sudo systemctl daemon-reload
sudo systemctl enable db-backup.service
sudo systemctl start db-backup.service

Output

$ systemctl status db-backup.service
● db-backup.service - Nightly DB Backup
   Loaded: loaded (/etc/systemd/system/db-backup.service; enabled)
   Active: active (running) since Thu 2025-07-24 10:00:00 IST

---

⚙️ Service Lifecycle Commands with systemctl

---

📋 View Targets & Units

• List all units:

sudo systemctl list-units --all

• Only active units:

sudo systemctl list-units

• Current default target (runlevel):

systemctl get-default

• Change target to headless server:

sudo systemctl set-default multi-user.target

---

📒 Logging & Troubleshooting with journalctl

systemd integrates closely with its logging tool journald.

Example Output:

Jul 24 10:05:01 appserver project-mercury.sh[12345]: Server started on port 8000
Jul 24 10:10:07 appserver project-mercury.sh[12345]: Connection timeout for user 10.0.0.5

---

🧠 Advanced Usage & Tips

• Use Restart=always for production-critical apps

• Add ExecStop=/path/to/cleanup.sh for graceful shutdowns

• Combine with systemd-timers for scheduling jobs (cron alternative)

• Use ProtectSystem=full, NoNewPrivileges=true, and PrivateTmp=true for security

• Run each service as a minimal-privilege user for isolation

Example with Advanced Options

[Service]
ExecStart=/usr/bin/myservice.sh
User=myserviceuser
Restart=always
Environment="ENV=prod"
ProtectSystem=full
PrivateTmp=true
NoNewPrivileges=true

---

🎓 Summary: Why systemd Matters for DevOps

✅ Centralized service management ✅ Resilience and self-healing via restarts ✅ Cleaner logging and diagnostics ✅ Easy to deploy custom services ✅ Reliable boot-time service orchestration

Whether you manage a Flask app, PostgreSQL server, or CI/CD runner, understanding systemd will help you:

• Automate deployments

• Improve system uptime

• Debug effectively