Sadservers Day 1: Hunting Down a Rogue Process Writing to bad.log

It’s Day 1 of my experiment with sadservers.com, a platform that calls itself “LeetCode for Linux”. As someone who loves troubleshooting, the idea of jumping into live sysadmin puzzles is both exciting and a little intimidating.

Today’s challenge dropped me straight into the thick of it, something’s flooding the /var/log/bad.log file, and my job is to find the culprit and stop it (without deleting the log).

---

Step 1: Spotting the Clue

My first check is always to look at the evidence. When I run:

tail -f /var/log/bad.log

Lines fly by, so some process is stubbornly writing to the file.

But there are hundreds of processes running, how do I pinpoint which one is filling up my disk?

---

Step 2: Tracking the Culprit

I remember that Linux exposes open file descriptors for each process under /proc/<pid>/fd/. If a process is attached to bad.log, I’ll see it there.

Time to let shell magic loose:

ls -l /proc/*/fd/* | grep bad

The command fights through some “No such file or directory” warnings, but finally coughs up a hit:

l-wx------ 1 admin admin 64 Aug 18 12:14 /proc/583/fd/3 -> /var/log/bad.log

Success! PID 583 is my suspect.

---

Step 3: Stopping the Flood

With the culprit identified, I go full sysadmin and terminate it:

kill -9 583

The file stops growing immediately. The server breathes a sigh of relief.

---

Alternate Solution: The fuser shortcut

For quick fixes, the fuser tool can instantly tell you which process is using a file:

fuser /var/log/bad.log

Grab the PID, check it with ps aux, and kill it. This shortcut is handy for troubleshooting, especially during high-pressure emergencies.

What I Learned Today:

Reflecting on Day 1, a few lessons stand out:

• The real problem is usually a runaway process, not the file itself.

• /proc/<pid>/fd/ gives x-ray vision into a running system, ideal for forensic work.

• Tools like fuser and lsof are essential for Linux detectives.

• Think before deleting, removing a log often just hides the problem.

---

Looking Ahead: More Sadservers Adventures

This was just Day 1. Sadservers offers plenty of unique, “real-life” Linux and DevOps problems. I’ll be exploring more challenges in the coming days and sharing both my failures and victory dances here.

If you love troubleshooting, or if you dread on-call war stories, follow along. Next up: maybe a misconfigured webserver, a sneaky cron job, or a stubborn firewall rule. If I get stuck, at least you’ll know what not to do.

Stay tuned for Day 2!

References:

• Stack Overflow

• sadservers.com