Capture Logs in Consul

To effectively troubleshoot issues in your Consul cluster, you must first identify whether the problem originates from the Consul agent or the service mesh. This distinction determines the logs and diagnostic tools required for efficient resolution. This guide outlines the steps to diagnose and collect relevant data for both scenarios.

1. Consul Agent-Related Issues

If the issue involves Consul agents (e.g., performance or operational problems), collect diagnostic data and logs from both the affected followers and leader using a Debug Bundle with command consul debug

The Debug Bundle includes:

• Logs: Captures all activities of the Consul follower or leader from startup.

• Profiling Data: Includes CPU, memory, goroutine, and trace profiling (pprof). To enable pprof, set enable_debug=true before running the consul debug command.

2. Service Mesh-Related Issues

For issues related to service mesh connectivity (e.g., service communication failures or routing problems), gather diagnostic data from the Envoy proxy instances involved in the data path (e.g., sidecars or gateways).

Collecting Envoy Diagnostics

Access the Envoy admin endpoint at 127.0.0.1:19000 to retrieve key diagnostic data:

• Listeners:

    curl 127.0.0.1:19000/listeners
  

• Clusters:

    curl 127.0.0.1:19000/clusters > clusters-<app-name>.txt
  

• Config Dump:

    curl 127.0.0.1:19000/config_dump?include_eds
  

• Stats:

    curl 127.0.0.1:19000/stats
  

Enabling Debug Logging for Envoy

To capture detailed logs from Envoy instances:

1. Enable debug logging on Envoy:

    curl -X POST 127.0.0.1:19000/logging?level=debug
   

2. Access logs in Kubernetes:

    kubectl logs -f <consul-mesh-gateway-pod> -c <envoy_container>
   

Tip: Replace <consul-mesh-gateway-pod> and <envoy_container> with the specific pod and container names in your environment.

Reference:

https://go.dev/doc/diagnostics