Introduction

Here in this article, We want to collect, process, and export metrics, logs, and traces from a Kubernetes cluster using OpenTelemetry—without needing additional exporters like Node Exporter, Kube State Metrics, Promtail, or Jaeger Agent.

🔹 Tools Used:
✅ OpenTelemetry Collector – Central processing for all telemetry data.
✅ Prometheus – Stores metrics.
✅ Loki – Stores logs.
✅ Jaeger – Stores traces.
✅ Grafana – Visualizes logs, metrics, and traces.

📌 Step 1: Install OpenTelemetry Collector in Kubernetes

1.1 Install OpenTelemetry Collector Using Helm

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector

🔹 This deploys the OpenTelemetry Collector in Kubernetes.

📌 Step 2: Configure OpenTelemetry Collector for Logs, Metrics, and Traces

2.1 Create OpenTelemetry Collector Config File

We will configure OpenTelemetry to:
✅ Scrape Kubernetes metrics (instead of using Node Exporter or Kube State Metrics).
✅ Collect logs from pods (instead of Promtail).
✅ Receive traces from applications (instead of Jaeger Agent).
✅ Send data to Prometheus, Loki, and Jaeger.

Create a file otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
      http:
  prometheus:
    config:
      scrape_configs:
        - job_name: "kubernetes"
          kubernetes_sd_configs:
            - role: pod
  filelog:
    include: ["/var/log/pods/*.log"]

processors:
  batch:
    timeout: 5s
  attributes:
    actions:
      - key: "environment"
        value: "production"
        action: insert

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"
  jaeger:
    endpoint: "jaeger:14250"
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, logging]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheus, logging]
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [loki, logging]

🔹 Metrics – Scrapes Kubernetes metrics instead of Kube State Metrics.
🔹 Logs – Reads logs from Kubernetes instead of Promtail.
🔹 Traces – Receives traces from apps instead of Jaeger Agent.

2.2 Apply OpenTelemetry Collector Configuration

kubectl apply -f otel-collector-config.yaml

📌 Step 3: Deploy Observability Backends

Now, we need storage for logs, metrics, and traces.

3.1 Deploy Jaeger for Traces

helm install jaeger jaegertracing/jaeger --set query.enabled=true

Verify Jaeger:

kubectl get pods -l app.kubernetes.io/name=jaeger

Access Jaeger UI:

kubectl port-forward svc/jaeger-query 16686:16686

Open http://localhost:16686 to see traces.

3.2 Deploy Prometheus for Metrics

helm install prometheus prometheus-community/prometheus

Verify Prometheus:

kubectl get pods -l app=prometheus

Access Prometheus:

kubectl port-forward svc/prometheus-server 9090:80

Open http://localhost:9090 to see metrics.

3.3 Deploy Loki for Logs

helm install loki grafana/loki-stack

Verify Loki:

kubectl get pods -l app=loki

Access Loki logs in Grafana.

📌 Step 4: Instrument Applications with OpenTelemetry

To collect logs, metrics, and traces, we need to instrument our applications.

4.1 OpenTelemetry Instrumentation for Python

Example Python Flask App instrumented with OpenTelemetry:

from opentelemetry import trace, metrics
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from flask import Flask

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

# Configure Tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
trace_exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(trace_exporter))

@app.route("/")
def home():
    with tracer.start_as_current_span("home-endpoint"):
        return "Hello, OpenTelemetry!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

4.2 Deploy Instrumented Application in Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-app
  labels:
    app: otel-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: otel-app
  template:
    metadata:
      labels:
        app: otel-app
    spec:
      containers:
      - name: otel-app
        image: my-instrumented-app:latest
        ports:
        - containerPort: 5000

Apply Deployment:

kubectl apply -f otel-app.yaml

📌 Step 5: Visualizing the Observability Data

Now that logs, metrics, and traces are being collected, you can visualize them in the respective tools.

5.1 View Traces in Jaeger

kubectl port-forward svc/jaeger-query 16686:16686

Visit http://localhost:16686 and search for traces.

5.2 View Metrics in Prometheus

kubectl port-forward svc/prometheus-server 9090:80

Visit http://localhost:9090 to query metrics.

5.3 View Logs in Loki (via Grafana)

kubectl port-forward svc/grafana 3000:80

Visit http://localhost:3000, add Loki as a data source, and query logs.

📌 Summary of the Setup

Conclusion

By following this guide, you have successfully set up a unified observability pipeline using OpenTelemetry in Kubernetes. Now, your system can collect, process, and analyze logs, metrics, and traces in one go!