To Monitor the Open Telemetry Collector with Prometheus and Grafana

Open Telemetry is an open-source observability framework created to provide standardized, processing, and exporting telemetry data, including metrics, traces, and logs, from applications and infrastructure. It was created by merging two earlier projects, Open Tracing and Open Census, under the Cloud Native Computing Foundation (CNCF).

Key Components of Open Telemetry

1. APIs and SDKs:
   Open Telemetry offers APIs for generating telemetry data and SDKs for processing and exporting that data. The APIs define standard interfaces for instrumentation, while the SDKs provide implementations for handling telemetry.

2. Exporters:
   Exporters are used to send collected telemetry data to various backends and monitoring systems like Prometheus, Jaeger, Zipkin, New Relic, Datadog, and others. Open Telemetry supports multiple exporters, allowing flexibility in how telemetry data is consumed.

3. Collectors:
   The Open Telemetry Collector is a standalone agent that can receive, process, and export telemetry data. It helps with centralizing data collection and processing before sending it to backends.

4. Semantic Conventions:
   Open Telemetry defines semantic conventions to standardize the way telemetry data is described. This helps in maintaining consistency across different telemetry data sources.

Features and Benefits

• Vendor-Agnostic:
  Open Telemetry provides a standard way to collect and process telemetry data that is independent of specific monitoring and observability vendors. This reduces vendor lock-in and increases flexibility in choosing backends.

• Unified Observability:
  By supporting metrics, traces, and logs, OpenTelemetry provides a comprehensive solution for observability, allowing developers and operators to gain insights into system performance and behavior.

• Wide Language Support:
  OpenTelemetry supports multiple programming languages, including Java, Python, Go, JavaScript, and more, making it suitable for diverse application ecosystems.

• Extensibility:
  The modular architecture of OpenTelemetry allows users to extend its capabilities by adding custom instrumentation or integrating with other tools and systems.

Use Cases

• Distributed Tracing:
  Track the flow of requests across services in a distributed system to identify bottlenecks and latency issues.

• Application Performance Monitoring (APM):
  Collect metrics and traces to monitor application performance and resource usage.

• Log Correlation:
  Correlate logs with traces and metrics to gain a comprehensive view of application behavior and diagnose issues more effectively.

• Microservices Observability:
  Monitor and debug microservices architectures by collecting and analyzing telemetry data from individual services.

How OpenTelemetry Works

1. Instrument Your Application:
   Use OpenTelemetry libraries to instrument your code and generate telemetry data.

2. Collect Telemetry Data:
   Use the OpenTelemetry Collector or SDK to collect and process telemetry data.

3. Export Data:
   Configure exporters to send data to observability backends or monitoring systems.

4. Analyze and Visualize:
   Use monitoring tools to analyze and visualize telemetry data to gain insights into system performance and health.

OpenTelemetry is becoming the industry standard for observability in cloud-native environments due to its flexibility, extensibility, and community support​ (HashiCorp Developer)​.

Let's start with Hands-On practical lab for Setup OpenTelemetry Collector on Ubuntu with Prometheus and Grafana.

Step1: Install Prometheus on Ubuntu Machine

Please refer the below page to install Prometheus on Ubuntu/Linux Box:

Step2: Install Grafana on Ubuntu Machine

Please refer the below page to install Prometheus on Ubuntu/Linux Box:

Step 3: Install Node Exporter on Ubuntu Machine:

Please refer the below page to install Prometheus on Ubuntu/Linux Box:

https://cloudhub2295.hashnode.dev/setup-prometheus-grafana-and-node-exporter-on-ubuntu

Step 4: Install OpenTelemetry Collector Agent on Ubuntu Machine

Please run the below commands to install Otel Collector for receiving the metrics.

We will update the system repository index by using the following command.

sudo apt update

Using below command we can download Open Telemetry, here we are downloading Open Telemetry 0.88.0 version, you use above link to download specific version.

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.88.0/otelcol-contrib_0.88.0_linux_amd64.deb

Installing a package with below command.

sudo dpkg -i otelcol-contrib_0.88.0_linux_amd64.deb

Move the given path and move the yaml file.

sudo cd /etc/otelcol-contrib
sudo mv config.yaml config_orig.yaml

open the below yaml file and add the code.

sudo vi /etc/otelcol-contrib/config.yaml

code

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  opencensus:
    endpoint: 0.0.0.0:55678

  # Collect own metrics
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otel-collector'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']
      - job_name: 'node-exporter'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:9100']


  #Data source: metrics
  hostmetrics:
    collection_interval: 10s
    scrapers:
      # CPU utilization metrics
      cpu:
      # Disk I/O metrics
      disk:
      # File System utilization metrics
      filesystem:
      # CPU load metrics
      load:
      # Memory utilization metrics
      memory:
      # Network interface I/O metrics & TCP connection metrics
      network:
      # Paging/Swap space utilization and I/O metrics
      paging:


processors:
  batch:

exporters:
  logging:
    verbosity: detailed

  prometheusremotewrite:
    endpoint: http://localhost:9090/api/v1/write

service:

  pipelines:

    metrics:
      receivers: [otlp, prometheus, hostmetrics]
      processors: [batch]
      exporters: [prometheusremotewrite]

  extensions: [health_check, pprof, zpages]

Change the ip address(localhost) to Prometheus server ip address in Prometheus .yaml file

After creating or modifying the service file, reload the systemd daemon to recognize the new service:

sudo systemctl daemon-reload

Start and Enable the OpenTelemetry Service.

sudo service otelcol-contrib start
sudo service otelcol-contrib enable

Verify that is OpenTelemetry running.

sudo service otelcol-contrib status

Now, You can access the Grafana service is ready to run, and we can access it from any web browser with Ip 3000

http://your_ip:3000

Here you can see Login page of Grafana now you will have to login with below Grafana default Username and Password.

Username – admin
password - admin

Now, you can see Home Dashboard page of Grafana.

Add the below Dashboard id in Grafana. in below image we have added for Prometheus , for Open Telemetry we have to add -1860 ,15983

Now provide the name and select the Prometheus Datasource and click on Import.

To Search all of the time series data points grouping by job count({name\=~".+"}) by (job)

Here, Successfully created all the steps for the Open Telemetry Collector with Prometheus and Grafana.