Debugging Insights: A New Lesson

The other day, I was solving another problem from the 100 Days DevOps Engineer by KodeKloud, and this is what I encountered.

The requirement was clear:

Only the load balancer (LBR) at 172.16.238.14 should be able to hit port 5000 on all of the app servers.
Everyone else → blocked.
And of course, it had to survive a reboot.

Here’s how it played out.

🔎 The Problem

Apache was running fine on all three app servers (stapp01, stapp02, stapp03) — but literally any host on the network could curl port 5000 and get in. No firewall, no restrictions. Just wide open.

That was the gap we had to close.

🛠 First Steps — Installing iptables

The Lab i was assigned with was using CentOS so i used these commands:

sudo yum install -y iptables iptables-services

⚡ Where Everything Became Chaos

This is where things got interesting. I added a rule to allow port 5000 from the load balancer… but it wasn’t working.

Turns out I had placed my ACCEPT rule below a general REJECT all. Rookie mistake. In iptables, rules are evaluated top to bottom. Once traffic hits a REJECT or DROP, that’s it — it never even sees the rules below.

Lesson learned.

Here’s what the final working rules looked like:

# Allow established/related connections
sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow ICMP (ping) from load balancer
sudo iptables -A INPUT -p icmp -s 172.16.238.14 -j ACCEPT

# Allow Apache port 5000 only from LBR (insert before REJECT)
sudo iptables -I INPUT 5 -p tcp --dport 5000 -s 172.16.238.14 -j ACCEPT

# Reject everything else
sudo iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

# Extra safeguard: Drop all other traffic to port 5000
sudo iptables -A INPUT -p tcp --dport 5000 -j DROP

💾 Making It Stick

Of course, rules in memory don’t survive reboots. So:

RHEL/CentOS:

sudo service iptables save
sudo systemctl enable iptables

✅ Testing the Fix

From the load balancer (stlb01):
```
  curl -v http://172.16.238.10:5000
```
I was able to get the html page.
From another host(It was jumphost in mycase):
```
  curl -v http://172.16.238.10:5000
```
❌ Blocked with No route to host.

That’s exactly what we wanted.

🚦 Key Takeaways From This

Rule order matters — top to bottom evaluation will burn you if you’re not careful.
Confirm listening ports before you go rule-crazy:
```
 sudo ss -tlnp | grep 5000
```
Test both allowed and blocked sources — don’t assume.
Persistence is key — otherwise you’ll be redoing this on the next reboot.
Logging helps — adding a quick log rule before DROP/REJECT can save you a lot of guessing.

📚 What Helped Me Learn

Red Hat iptables docs
Ubuntu iptables how-to
Googling and Little ChatGPT

At the end of it, I locked down Apache’s port 5000 so only the load balancer could talk to it, and in the process I walked away with a much better understanding of just how picky iptables can be about rule order.

Sometimes the fix isn’t about the tool itself, but about how you structure the rules.

Another Debugging Lesson