Directory Discovery with Gobuster: Legal Hacking That Feels Like Trespassing

🔎 What Is Gobuster?

Gobuster is a tool that helps you **find hidden folders and files** on a web server — things like `/admin/`, `/backup/`, `/test/`, or even `.git/`.

These paths may not show up on the website itself, but they still exist. And sometimes, they're not protected at all.

Instead of checking manually, Gobuster uses a **wordlist** (a file with common directory names) and tests each one against a URL.

Think of it like trying every key on a massive keyring and watching which doors open.

---

🧪 My First Gobuster Scan

I ran a basic directory scan against a test site:

gobuster dir -u http://www.example.com -w /usr/share/wordlists/dirb/common.txt

What it does:

• `dir` tells Gobuster to do directory brute-forcing.

• `-u` sets the target URL.

• `-w` specifies the wordlist to use.

After a few seconds, it returned this:

```

/index.html (Status: 200) [Size: 1256]

Progress: 4614 / 4615 (99.98%)

```

Here I only found a single index.html page. The file common.txt contains a list of 4615 words. With a bigger list I might have obtained more directories.

Next I tried scanme.nmap.org:

gobuster dir -u http://scanme.nmap.org -w /usr/share/wordlists/dirb/common.txt

Here I got a lot more directories :

/.hta (Status: 403) [Size: 286]

/.htaccess (Status: 403) [Size: 291]

/.htpasswd (Status: 403) [Size: 291]

/.svn/entries (Status: 403) [Size: 294]

/.svn (Status: 301) [Size: 316] [--> http://scanme.nmap.org/.svn/]

/favicon.ico (Status: 403) [Size: 293]

/images (Status: 301) [Size: 318] [--> http://scanme.nmap.org/images/]

/index (Status: 200) [Size: 6974]

/index.html (Status: 200) [Size: 6974]

/server-status (Status: 403) [Size: 295]

/shared (Status: 301) [Size: 318] [--> http://scanme.nmap.org/shared/]

Progress: 4614 / 4615 (99.98%)

Status code 200 indicates the path was found and returned, 301 indicates a redirect, 403 indicates an existent but unauthorized path.

---

🎯 Why Use Gobuster?

• You can find hidden files or folders not linked on the site

• You might discover forgotten admin panels or dev folders

• It helps map out the attack surface before you even think of exploiting anything

---

🧠 What I Learned

• Wordlist choice matters. A longer wordlist means more thorough scanning — but takes longer.

• Status codes tell stories. `200`, `403`, and even `401` all mean something is *there*.

• This is noisy. You’re making a lot of requests — which could be logged. In real engagements, stealth and ethics matter.

---

💡 Pro Tips I Found Useful

• Use `-x` to look for file extensions:

 gobuster dir -u https://example.com -w common.txt -x php,html,txt

• Add `-t` to control thread count (speed vs stealth):

gobuster dir -u https://example.com -w wordlist.txt -t 20

• Use small, focused wordlists for quick tests like `common.txt`, then go deeper with bigger ones like `directory-list-2.3-medium.txt`.

---

🔁 TL;DR

Gobuster makes directory discovery faster and easier. Even as a beginner, it gives you a real sense of how websites are structured behind the scenes — and how much they accidentally reveal.

This is one of the first tools where I felt like I was actively exploring rather than just watching from a distance.

Just remember: use it only where you have permission. Otherwise, it’s not recon — it’s illegal.