🌱 Cultivating Unique service.instance.id on NGINX Ingress with OpenTelemetry

A task to set a unique service.instance.id
on NGINX Ingress Controller using the OpenTelemetry should be simple, right? But as it turned out, the ingress controller doesn't expose all of NGINX’s OTel knobs out of the box, so I had to roll my own tweak garden.
Why should I care?
Without unique instance IDs, your tracing data looks like a tangled tangleweed—hard to trace, difficult to debug, and completely undermines what we’re trying to do with observability.
The solution
Part 1 - Nginx Ingress Controller - Helm Chart
To get POD_UID
injected into each trace, here’s the minimal yet powerful config snippet I landed on:
- Set the POD_UID environment variable, once NGINX Ingress Controller set by default only the POD_NAME and POD_NAMESPACE.
extraEnvs:
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- Set the POD_UID as span-attribute for instance.
controller:
config:
main-snippet: |
env POD_UID;
server-snippet: |
set $pod_uid "unknown";
access_by_lua_block {
ngx.var.pod_uid = os.getenv("POD_UID") or "unknown"
}
opentelemetry_attribute service.instance.id $pod_uid;
Part 2 - OpenTelemetry Collector Config
- Configure the transform processor to set the resource-attribute.
processors:
transform:
error_mode: ignore
trace_statements:
- set(resource.attributes["service.instance.id"], span.attributes["service.instance.id"])
- delete_key(span.attributes, "service.instance.id")
How it works:
- Explicitly exposes the
POD_UID
environment variables so NGINX can see them. - Initializes a default
$pod_uid
—my fail-safe in case the env var goes missing. - Uses Lua to pull in the real
POD_UID
. - Finally, sets the OpenTelemetry attribute
service.instance.id
to match the actual UID. - Through OpenTelemetry Collector capture the
service.instance.id
from span-attributes and set it on resource-attributes.
Benefits to your telemetry:
Without unique service.instance.id | With unique service.instance.id |
Pods share the same instance ID, causing confusion in telemetry. | Each pod is individually identifiable. |
Difficult to isolate errors and debug effectively. | Allows precise tracing to individual pods. |
Limited visibility into pod-specific issues. | Enables accurate root-cause analysis. |
Final result
By implementing this solution, each NGINX ingress pod is clearly distinguishable in tracing data. This improves observability significantly by providing accurate, pod-specific telemetry, facilitating precise troubleshooting and diagnostics.
Subscribe to my newsletter
Read articles from Yuri Oliveira Sá directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
