Debugging masterclass!

Recently, I joined KodeKloud’s 100 Days DevOps Engineer Bootcamp, and I want to share an experience I had with a question and the insights I gained from it.

The task was troubleshooting a MariaDB connection issue for an app, and things were already looking a bit suspicious. The error logs pointed me straight to /var/lib/mysql — that’s where the MySQL folder was supposed to be. But when I checked, there was no mysql directory at all.

Instead? There was a folder called /mysqld.

Now, in my head, this made perfect sense. “Oh, this must be the new thing. Maybe in recent versions they switched the folder name. And hey, mysqld sounds like some daemon process for MySQL anyway, right?”

So I thought maybe the ownership was wrong. I changed the permissions… and the app still refused to work. I tinkered a bit more, got stuck, tinkered again — you know how it goes.

In the end, the solution was a painfully simple:

mv mysqld/ /var/lib/mysql

That was it. lot of overthinking, gone.

The real flaw? My assumptions. I thought I knew what mysqld meant and convinced myself it was a version change — instead of doing the obvious: checking the documentation or searching online to confirm.

So I thought I should have a proper flow for debugging services and applications so that the next time I encounter something like this, it will be a cakewalk. Below are some troubleshooting techniques that I distilled, which follow a directed approach to solve the problem and pinpoint the error.

---

Digging Into Services with systemctl and Friends:

Once I figured out the root cause, I realized I could have saved a lot of time if I had just leaned more on the tools we already have for debugging services.

Here’s what I tend to use now, with MariaDB as an example:

systemctl status mariadb.service        # Quick health check
journalctl -xeu mariadb.service         # Get detailed logs
systemctl --help                        # Explore more options
systemctl list-units --type=service     # See all running services

---

Useful Debugging Concepts I Wish I Used Sooner

1. Application Logs → Usually in /var/log/<application-name>. But keep in mind, some apps have their own custom log directories.

2. Configuration Files → Typically in /etc/<application-name>.

   • /etc/systemd/system/ is where you’ll find service unit files for systemd services.

3. Shared Libraries & Modules → /lib/ contains core system libraries and kernel modules.

4. Executable Binaries → /usr/bin and /usr/sbin store the actual executables for services and apps.

---

Traversing and Reading the Right Files

One thing that caught me early on: ROOT sees different files than a regular USER. So if you’re debugging, make sure you’re checking as the right user.

I now make it a habit to:

• Use grep, cat, less, and tail to read through config and log files.

• Run ps aux to see if processes are actually alive.

• Combine systemctl and journalctl for quick service checks and logs.

• For networking issues, use netstat or ss to confirm open ports.

• Don’t forget to check for sneaky cronjobs that might be interfering.

---

Using systemctl Commands to Our Advantage

My Go-To Commands (Now That I Know Better)

systemctl – Manage services:

systemctl list-units --type=service
systemctl status <service_name>
systemctl start <service_name>
systemctl stop <service_name>
systemctl restart <service_name>
systemctl enable <service_name>

journalctl – View logs:

journalctl -u <service_name>
journalctl -f

netstat / ss – Network checks:

netstat -tuln
ss -tuln

df – Disk space:

df -h

top / htop – Process monitoring:

top
htop

strace – Trace system calls:

strace -p <pid>

lsof – List open files:

lsof -p <pid>

---

Looking back, this whole thing was a reminder for me: sometimes the fastest path to fixing an issue is not assuming you already know the answer — it’s slowing down, checking the basics, and using the tools at your disposal.

Next time I run into a problem like this, I’m starting with systemctl and journalctl then go through the log files and binaries before I let my imagination run wild.

I understand that this topic isn't strictly related to DevOps, yet it is deeply connected to it. I believe DevOps isn't just about implementation; it's a methodology to practice.