Debugging Kubernetes Networking: Why can’t my service be reached

It started with a simple deployment. The app was up, pods were running, but the service just wouldn’t respond. Requests vanished into thin air and users saw... nothing. This wasn’t the first time, so I rolled up my sleeves for some hands-on Kubernetes networking debugging.

Where Did My Service Go?

Networking in Kubernetes feels magical, until it doesn’t. When a service isn’t reachable, there’s usually one of three culprits: configuration, DNS, or connectivity. The key is to check each layer methodically.

Step 1: Service and Endpoint Checks

Begin with the basics. Run:

kubectl get service
kubectl describe service <service-name>
kubectl get endpoints <service-name>

This tells you whether your service exists, how it’s set up, and which pods (if any) are behind it. If endpoints are missing, pods might not be labelled correctly or failing their readiness probes.

Step 2: DNS Resolution

A lot of service-to-service communication inside Kubernetes happens by DNS name. To check DNS:

kubectl exec -it <any-pod> -- nslookup <service-name>
kubectl exec -it <any-pod> -- dig <service-name>

No DNS result? Double check CoreDNS is running, and look at its logs for errors. Bad DNS config can make a working service look invisible.

Step 3: Connectivity and Network Policies

With endpoints and DNS looking good, check actual network paths:

kubectl exec -it <any-pod> -- curl <service-name>:<port>
kubectl exec -it <any-pod> -- telnet <service-name> <port>

If these fail, something in the network is blocking packets. Inspect your network policies:

kubectl get networkpolicy
kubectl describe networkpolicy <policy-name>

Network policies can restrict traffic between namespaces or groups of pods. Loosen them to test, then tighten up with explicit rules.

Step 4: Debug with Netshoot and Tcpdump

Sometimes you need to peek into the wires. The netshoot container is packed with tools for low-level debugging (like tcpdump, ngrep, and more) and lets you capture and inspect traffic between pods. Run a debug pod with:

kubectl run netshoot --rm -it --image nicolaka/netshoot -- bash

From there, use tools like tcpdump or ss to hunt for traffic flowing in and out.

What Caused It This Time?

In my case, a network policy blocked requests from one namespace to another. It took an hour of poking at endpoints, running nslookup, and finally peering at tcpdump results to see the attempted but dropped packets. The fix was to update the policy and reapply labels where needed.

The Takeaway

Kubernetes networking is powerful, but it rewards careful, step-by-step debugging. Stick to the basics first, dig deeper if everything seems right, and use the tools the community builds for these exact situations. The next time your service vanishes, start with endpoints and policies, you’ll find your culprit sooner.