Opentelemetry Bonus Tricks

Convert Kong logs to OTLP format

I faced issues when I worked on logs and the OpenTelemetry protocol.
Kong doesn't provide logs in this format, and to enable Log for the span feature in Grafana, we need to collect logs in the OTLP format.

I used Fluentbit to perform these tasks.

$ cd && mkdir -p ~/opentelemetry_demo/fluentbit && cd ~/opentelemetry_demo/fluentbit

Let's create FluentBit config file

$ cat << EOF >> config.conf
[INPUT]
    name http
    listen 0.0.0.0
    port 8888

[FILTER]
    Name    lua
    Match   kong
    code    function inline_filter(tag, timestamp, record)record.trace_id = record.trace_id.w3c; return 1, timestamp, record end
    call    inline_filter

[OUTPUT]
    name stdout
    match  *

[OUTPUT]
    Name                 opentelemetry
    Match                kong
    Host                 otelco
    Port                 4318
    Metrics_uri          /v1/metrics
    Logs_uri             /v1/logs
    Traces_uri           /v1/traces
    Log_response_payload True
    Tls                  Off
    Tls.verify           Off
    logs_trace_id_message_key trace_id
    logs_span_id_message_key span_id
EOF

I couldn't collect or create a specific service_name on logs emits by Kong. So, this following did the trick but it's not really an happy path.

I added a processors -> resource -> attributes -> action: upsert on OTEL collector for logging stack.
If a service_name is missing, OpenTelemetry Collector will create a default one.

If anyone has a better solution, please make a comment.

$ cat << EOF >> ~/opentelemetry_demo/otelcol/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otel-collector'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']

processors:
  batch:
  resource:
    attributes:
      - action: upsert
        key: service.name
        value: kong-dev

exporters:
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlphttp:
    endpoint: http://loki:3100/otlp
  prometheusremotewrite:
    endpoint: http://mimir:8080/api/v1/push

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlphttp]
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [prometheusremotewrite]
EOF

Enable Kong HTTP-Log plugin

$ curl -X POST http://localhost:8001/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
  "name": "http-log",
  "config": {
    "http_endpoint": "http://fluentbit:8888/kong",
    "method": "POST",
    "timeout": 1000,
    "keepalive": 1000,
    "flush_timeout": 2,
    "retry_count": 15
  }
}
    '

Now, create the FluentBit container

$ docker run \
  --name=fluentbit \
  --hostname=fluentbit \
  --network=lgtm \
  -p 8888:8888 \
  -v $(pwd)/config.conf:/etc/fluentbit/config.conf \
  cr.fluentbit.io/fluent/fluent-bit \
  --config=/etc/fluentbit/config.conf

Finally, you need to configure the Tempo datasource in Grafana by setting Trace to logs this way :

Now, we can fetch logs from a Kong trace ID.