Log to Splunk with Serilog and .NET

Splunk provides an HTTP endpoint, known as an Event Collector, which can be used to POST log messages directly to the Splunk database. In this article we see how Serilog can be used in an ASP.NET Core Web API to write messages into Splunk using the HTTP Event Collector.

In enterprise scenarios the Splunk instance used in a production environment will be managed by another team, but ideally we would run a similar setup for our local development so that we can ensure our logging is fit for purpose and that we know how to use the production debug tools when required.

Consequently I will also show how to setup a local Splunk instance with the HTTP Collector enabled illustrating a possible local development scenario.

Setting Up Serilog in .NET Core

Serilog is a popular logging framework which has been available in .NET for many years. To get up and running on .NET Core 8 there is a Serilog.AspNetCore package which includes everything required for ASP.NET Web APIs. Once you’ve installed this package, update the Program.cs of your Web API with the following code:

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .CreateLogger();

try
{
    var builder = WebApplication.CreateBuilder(args);
    builder.Host.UseSerilog(); 

    //
    // Add services etc...
    //
}
catch (Exception ex)
{
    Log.Fatal(ex, "Application terminated unexpectedly");
}
finally
{
    Log.CloseAndFlush();
}

It’s worth noting that Serilog is an opinionated library which completely replaces the whole logging infrastructure as described in the blog post Setting up Serilog in ASP.NET Core 3:

Serilog is also independent of .NET Core and most of the framework infrastructure (including configuration and dependency injection), making it well-suited to collecting and recording problems with starting up the framework itself

See the Serilog documentation for more information.

Setting Up Serilog Http Event Collector Middleware

Serilog uses a "sink" as an abstraction to model the destination where log events are sent for storage, processing, or analysis. The package Serilog.Sinks.Splunk installs the sink for the Splunk HTTP Event Collector. Once installed we need to add an extra line to the previous LoggerConfiguration setup shown above:

 Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.EventCollector("http://localhost:8088/services/collector", "f7086918-ff5f-4471-a534-65d3a8d4b97c")
    .CreateLogger();

The GUID is known as the HEC token and which will be set up in the next section.

Setting up Splunk 8.2

Although Splunk is provided in a Docker container and should be straightforward to setup, I have found it to be a bit buggy. Start by copying the following YML to a docker-compose.yml file:

version: "3"

services:
  splunk:
    image: splunk/splunk:8.2
    container_name: splunk02
    environment:
#     - SPLUNK_ADMIN=admin
      - SPLUNK_PASSWORD=Password1!
      - SPLUNK_START_ARGS="--accept-license"
    volumes:
      - .\data\var:/opt/splunk/var
      - .\data\etc:/opt/splunk/etc
      - .\default.yml:/tmp/defaults/default.yml
    ports:
      - 8008:8000
      - 8088:8088

Next save the following into a default.yml file. This file has the settings that enable the HTTP Event Collector in the Splunk instance. The GUID shown is the HEC token used in the code excerpt above.

splunk:
  hec:
    enable: True
    ssl: false
    port: 8088
    token: f7086918-ff5f-4471-a534-65d3a8d4b97c

Running docker compose up with the above files caused the initial Splunk install to error out with the following message:

The fix ended up being to edit one of the files created during the first run of docker compose up. Find the file data\etc\splunk-launch.conf and add the key OPTIMISTIC_ABOUT_FILE_LOCKING setting it to 1 as shown below:

Run docker compose up again and Splunk should start correctly this time. Once up and running, you can test the Splunk installation using curl. Note that the GUID set in the default.yml file becomes the HTTP auth token for the request:

 curl -k http://localhost:8088/services/collector -H "Authorization: Splunk f7086918-ff5f-4471-a534-65d3a8d4b97c" -d '{"event": "This is a test splunk log message"}'

And if we navigate to the portal at localhost:8008 login with user admin and password Password1! then go to the “Search & Reporting” menu in the side bar we can search for the test message:

.NET Logs In Splunk

If we run the ASP.NET Web API with Serilog installed as outlined above, then the logs will get written through to the Splunk instance using the HTTP Event Collector. In the screenshot below we can see the logs loading in Splunk that were written using this mechanism.

Summary

In this article, we explored how to integrate Serilog with an ASP.NET Core Web API to send logs to Splunk using the HTTP Event Collector. We covered the setup of Serilog, including the necessary packages and configuration, and demonstrated how to configure a local Splunk instance with the HTTP Event Collector enabled. This integration not only enhances the logging capabilities of .NET applications but also provides a robust solution for analysing and managing log data within Splunk.

I found the following resources useful for getting up to speed with Splunk:

Using Serilog and .NET to Write Logs to Splunk HTTP Event Collector