Troubleshooting Pod-to-service Communication in k8s: The Proxy Trap

Life as a DevOps engineer gets interesting when a “simple” microservice-to-microservice call in Kubernetes…just refuses to work.

Recently, I ran into a problem you might recognize if you work with microservices on k8s. Two microservices, both running in a namespace called app, just wouldn’t talk to each other via a Kubernetes ClusterIP Service. Logging into one application pod, I kept seeing errors like:

tcp 172.20.186.216:3939: connect: connection refused
time="2024-01-31T06:33:01.618Z" level=error msg="http: proxy error: dial tcp 172.20.186.216:3939: connect: connection refused"
192.168.208.226 - - [31/Jan/2024:06:32:58 +0000] "GET /content/abcd/ HTTP/1.1" 502 0
time="2024-01-31T06:33:01.619Z" level=info msg="Connected to session server [http://service-a.app:50734/](http://service-a.app:50734/)"

The Usual Suspects: Ping and Service Connectivity

First steps in troubleshooting? Ping the pod, then ping the service IP. Strangely, both worked!
Next, I checked environment variables for proxy settings:

env | grep -i proxy

Everything looked fine:

no_proxy=192.168.0.0/16,172.20.0.0/16,.mydomain.com,localhost,127.0.0.1

So it should’ve excluded both the pod and service addresses from proxying. Or so I thought.

The Clever Culprit: DNS Name and Proxy

I ran a curl command inside the pod:

curl -ivk http://<servicename.namespace:port>

And noticed—the container was still routing the request through the proxy!
Why? Here’s the trick: The application was connecting to the service using its DNS name (servicename.namespace), not its ClusterIP.

The NO_PROXY variable only contained the raw IP network CIDR ranges, not the DNS domain used by services within the namespace.

The Solution: Update NO_PROXY for Your Namespace

What finally did the trick? Adding .app (the namespace as a DNS suffix) to NO_PROXY made everything work.

Example pod spec with the correct environment:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: app
spec:
  containers:
  - name: mycontainer
    image: myimage:latest
    env:
      - name: HTTP_PROXY
        value: http://proxy.corp.domain:8080
      - name: HTTPS_PROXY
        value: https://secureproxy.corp.domain:8080
      - name: NO_PROXY
        value: 192.168.0.0/16,172.20.0.0/16,*.mydomain.com,localhost,127.0.0.1,*.app

With *.app in NO_PROXY, Kubernetes service DNS calls are excluded from proxy interception—and the services communicate as intended.

Key Takeaways

• Always add your namespace (and/or service domain suffix) to NO_PROXY when using proxies in Kubernetes pods.

• Proxy variables prevent endless debugging of mysterious 502s and connection refused errors.

• Don’t just rely on IP/CIDR in NO_PROXY; DNS-based service names need coverage too.