Using OpenTelemetry, Prometheus, and Grafana to Instrument .Net App Metrics

Instrument your .NET application by using Prometheus for metrics collection and Grafana for creating insightful dashboards.

Understanding the Components:

• OpenTelemetry: A set of tools, APIs, and SDKs for collecting and exporting telemetry data (metrics, logs, and traces).

• Prometheus: An open-source monitoring system for collecting and aggregating metrics.

• Grafana: An open-source analytics and observability platform for visualizing metrics, logs, and traces.

• AWS Account with EC2 Instance.

• .NET SDK, Docker, Prometheus and Grafana installed.

Configure the .NET Application

Update the package list to configure the .NET application..

sudo apt update

Install the .NET SDK on your Ubuntu server.

sudo apt install -y dotnet-sdk-8.0

Clone our sample .NET application from the Opentelemetry GitHub repository using the given command.

git clone https://github.com/Ankita2295/opentelemetry-prometheus-for-dotnet-app

Navigate to the project directory.

cd opentelemetry-prometheus-for-dotnet-app

Open the Program.cs file

vi Program.cs

Add the following line to the top of your Program.cs file.

using System.Globalization; and using Microsoft.AspNetCore.Mvc;

using System.Globalization;
using System.Diagnostics.Metrics;
using Microsoft.AspNetCore.Mvc;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Instrumentation.AspNetCore;
using OpenTelemetry.Instrumentation.Http;
using Prometheus;


var builder = WebApplication.CreateBuilder(args);

const string serviceName = "roll-dice";


// Add services to the container.
builder.Services.AddControllers();

// Create a custom meter for the API
var meter = new Meter("roll-dice.Metrics", "1.0");


var httpRequestCounter = meter.CreateCounter<long>("http_requests_total", description: "Total number of HTTP requests");


builder.Logging.AddOpenTelemetry(options =>
{
    options
        .SetResourceBuilder(
            ResourceBuilder.CreateDefault()
                .AddService(serviceName))
        .AddConsoleExporter();
});

// Configure OpenTelemetry
builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
    {
        tracerProviderBuilder
            .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("roll-dice"))
            .AddAspNetCoreInstrumentation() // Tracks incoming HTTP requests
            .AddHttpClientInstrumentation()
            .AddConsoleExporter(); // Optional: For debugging
    })
    .WithMetrics(meterProviderBuilder =>
    {
        meterProviderBuilder
            .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("roll-dice"))
            .AddAspNetCoreInstrumentation() // Tracks incoming HTTP request metrics
            .AddHttpClientInstrumentation()
            .AddPrometheusExporter(); // Expose metrics to Prometheus
    });

// Add middleware to count HTTP requests
var app = builder.Build();

// Use the Prometheus middleware to expose the /metrics endpoint.
app.UseRouting();

app.UseHttpMetrics(); // Middleware for collecting HTTP metrics

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
    endpoints.MapMetrics(); // Expose the /metrics endpoint for Prometheus scraping
});



string HandleRollDice([FromServices]ILogger<Program> logger, string? player)
{
    var result = RollDice();

    if (string.IsNullOrEmpty(player))
    {
        logger.LogInformation("Anonymous player is rolling the dice: {result}", result);
    }
    else
    {
        logger.LogInformation("{player} is rolling the dice: {result}", player, result);
    }

    return result.ToString(CultureInfo.InvariantCulture);
}

int RollDice()
{
    return Random.Shared.Next(1, 7);
}

app.MapGet("/rolldice/{player?}", HandleRollDice);

app.Run();

Explanation of the OpenTelemetry Instrumentation code for .Net Application

Imports

These lines import the required namespaces for OpenTelemetry, ASP.NET Core, metrics, and Prometheus, offering classes and methods to build the application's telemetry and HTTP features.

2. Creating the Web Application

WebApplication.CreateBuilder(args) initializes the ASP.NET Core application builder, while serviceName specifies the service's name for identifying the source of traces and metrics in telemetry.

3. Adding Services to the Container

Adds controller support to the application, enabling the MVC framework to handle HTTP requests through controller methods.

4. Creating a Custom Meter for Metrics

Creates a custom Meter named roll-dice.Metrics to track metrics for the application, with a counter metric called http_requests_total to monitor the total number of HTTP requests.

5. Configuring OpenTelemetry for Logging

Configures OpenTelemetry to manage logging for the service named roll-dice, and AddConsoleExporter() enables console logging to display telemetry data for debugging.

6. Configuring OpenTelemetry for Tracing and Metrics

Configures OpenTelemetry to instrument HTTP requests and client interactions for tracing, exporting data to the console, and to track HTTP request metrics for Prometheus exposure using the AddPrometheusExporter() method.

7. Building the Application and Configuring Middleware

app.UseRouting() sets up request routing, while app.UseHttpMetrics() integrates Prometheus HTTP metrics to automatically track details about incoming HTTP requests.

8. Exposing the /metrics Endpoint

Maps endpoints for the controllers.MapMetrics() exposes the /metrics endpoint, allowing Prometheus to scrape metric data from the application.

9. Handling the /rolldice Endpoint

The HandleRollDice method processes requests to the /rolldice endpoint by rolling a dice with the RollDice() method, logging the result, and recording the player's name if provided, or noting an anonymous player if not.

The RollDice method generates a random number between 1 and 6 to simulate a dice roll, and app.MapGet("/rolldice/{player?}", HandleRollDice); connects the /rolldice endpoint to the HandleRollDice method, allowing access to the dice-rolling logic through HTTP requests.

10. Running the Application launches the web app to listen for incoming HTTP requests, and you open the MyPrometheusApp.csproj file.

vi MyPrometheusApp.csproj

Update the packages to the latest versions.

“OpenTelemetry.Instrumentation.AspNetCore” Version=”1.7.0″ and “OpenTelemetry.Instrumentation.Http” Version=”1.7.0″

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="8.0.8" />
    <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.9.0" />
    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.9.0-beta.2" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.6.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.7.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.7.0" />
    <PackageReference Include="prometheus-net.AspNetCore" Version="6.0.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.4.0" />
  </ItemGroup>

</Project>

The project file configures the .NET web application to use modern observability tools and web standards.

1. The project is defined to use the Microsoft.NET.Sdk.Web SDK, tailored for building ASP.NET Core web applications with all required tools and dependencies.

2. The PropertyGroup section includes several important settings: TargetFramework specifies the .NET version the application targets, which is .NET 8.0, defining the runtime and library versions the application will use. Nullable is set to enable, activating nullable reference types in C#, which helps handle null values explicitly, reducing null reference errors and increasing code safety. ImplicitUsings is also set to enable, automatically including common using directives, so you don't have to write them at the top of each file, simplifying the code by implicitly including commonly used namespaces.

3. ItemGroup Section: This section lists all the external dependencies (NuGet packages) that the project requires.

Set Up Docker on Ubuntu

Use the following command to install Docker on your Ubuntu server.

sudo apt install -y docker.io

sStart and enable the Docker service.

sudo systemctl start docker
sudo systemctl enable docker

Set Up Prometheus Using Docker

Let’s get the prometheus using Docker.

sudo docker pull prom/prometheus

Next open the prometheus.yml

nano prometheus.yml

Change the targets, ensuring you use your instance's public IP address.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'MyPrometheusApp'
    scrape_interval: 5s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['<EC2-instance-IP>:8080']

Build and Run the Application

To compile and run your application, use the command to build it.

dotnet build

Run the application.

dotnet run

This will start the application and expose the /metrics endpoint for Prometheus to collect data.

You can test your application by visiting the endpoint that provides the HTTP metrics, such as http://<Instance-IP>:8080/metrics.

http://<EC2-Instance-IP>:8080/metrics

You can run application rolldice by using http://<Instance-IP>:8080/rolldice.

This endpoint would simulate a dice roll and return the result of that roll.

Navigate to the project directory.

cd opentelemetry-prometheus-for-dotnet-app

Run the Prometheus container.

sudo docker run -p 9090:9090 -v ~/opentelemetry-prometheus-for-dotnet-app/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Once the container is running, you can access the Prometheus web interface by visiting http://<Instance-IP>:9090.

To verify Prometheus is scraping metrics from your .NET application, go to the "Targets" page in the Prometheus UI, click "Status" in the menu, then "Targets," and ensure your job MyPrometheusApp is listed and marked as "UP."

Set Up Grafana Using Docker

Run the following command to download the Grafana Docker image.

sudo docker pull grafana/grafana

Lets run the Grafana container.

sudo docker run -d -p 3000:3000 --name=grafana grafana/grafana

Grafana can be accessed at http://<Instance-IP>:3000.

The default Grafana login credentials are Username: admin and Password: admin, and you will be prompted to change the password after your first login, with the option to change it or skip.

Configure Prometheus as a Data Source in Grafana

After logging in to Grafana, you’ll need to add Prometheus as a data source.

• Click on “Connections” →“Data source”

Search for prometheus in search bar and select it as a data source. Then click on Add new data source.

Set the URL to http://<Instance-IP>:9090, replacing <Instance-IP> with your EC2 instance's IP address, leave other settings as default, click “Save & Test” to check Grafana's connection to Prometheus, and then click the “Plus” icon in the sidebar to select “Dashboard” and “Add visualization.””

In the “Query” section, select the Prometheus data source. Enter a Prometheus query to retrieve metrics from your . NET application, such as

http_requests_received_total

Choose a visualization type like “Time series,” “Graph,” or “Gauge” to best represent your data, and save the dashboard by clicking “Save Dashboard” at the top and giving it an appropriate title.