Article 1: strace — The X-Ray Vision for Debugging Embedded Linux

🚀 Introduction

In embedded Linux development, things often go wrong in silence. A binary exits with no error, a script hangs mid-way, or an init service fails on boot — and all you get is… nothing. No logs. No helpful messages. No clue.

This is where strace becomes your best friend.

---

🔍 What is strace?

strace is a userspace utility that traces system calls made by a process. It shows how your program interacts with the kernel — opening files, accessing memory, communicating via sockets, forking processes, and more.

It’s like watching your binary whisper to the OS.

For embedded systems, where visibility is often limited and conventional debugging tools (like GDB or IDEs) are unavailable, strace can be a lifesaver.

---

🛠️ Why It Matters in Embedded Linux

• Diagnosing Boot Failures: See what the early init services or binaries do (or fail to do).

• Finding Missing Dependencies: Spot missing shared libraries or config files.

• Debugging Silent Crashes: Learn what syscall failed before your app bailed.

• Monitoring Resource Use: Know how much your process opens, reads, and writes.

• No Kernel Mod Needed: Doesn’t require rebuilding anything or adding kernel modules.

---

⚙️ Basic Usage

strace ./your_binary

This command prints every syscall made by your_binary, along with arguments and return values.

Example output:

open("/etc/config.txt", O_RDONLY) = -1 ENOENT (No such file or directory)

That alone tells you what went wrong — the file was expected but not found.

---

📦 Tip: Installing strace on Embedded Devices

On Debian-based images:

apt-get update && apt-get install strace

On Buildroot:

menuconfig → Target packages → Debugging tools → strace

Or include it via your BR2_PACKAGE_STRACE=y config line.

---

🧪 Real-World Use Case: Binary Not Working on Target

You compiled a simple binary. It works on your dev machine but fails silently on the target.

strace ./binary

And you see:

execve("./binary", ["./binary"], [/* 14 vars */]) = -1 ENOENT

Turns out the shared libraries aren’t found. Use:

ldd ./binary

to confirm, or

strace -e trace=execve ./binary

to narrow scope.

---

🎯 Filtering Output

strace output can be overwhelming. Use filters:

• Trace only file access:

strace -e trace=file ./app

• Trace only network calls:

strace -e trace=network ./server

• Log to a file:

strace -o trace.log ./app

• Attach to running PID:

strace -p 1234

---

🧰 Embedded Debug Trick: Trace Init on Boot

Got an embedded board where init fails mysteriously?

1. Replace init with a script:

#!/bin/sh
strace -o /tmp/init_trace.log /sbin/init.real

2. Repack your image or modify inittab.

3. Boot and grab the log over SSH or serial.

---

🧠 Summary

strace is your first tool in building intuition about Linux system behavior. It’s simple, fast, powerful, and requires no kernel access. In embedded development, where loggers and debuggers are stripped out to save space, strace brings visibility back.

---

✅ Recap: When to Use strace

---

Next up in the series: lsof — Finding What Your System Is Really Using