Monitoring and Logging in ASP.NET Core Web APIs: Using Application Insights and ELK Stack

Opaluwa Emidowo-ojoOpaluwa Emidowo-ojo
11 min read

Welcome to the 13th installment of our series, glad to have you back! Today's technical tutorial is a guide to tracking and logging in ASP.NET Core Web APIs using application insights, and the Elasticsearch, Logstash, and Kibana stack.

When you build web APIs, it's important to keep an eye on how they are performing and to understand what's happening behind the scenes. This is where monitoring and logging come in.

Pre-requisites

To fully benefit from this article, readers should have the following prerequisites:

  • Basic Knowledge of ASP.NET Core

    • Familiarity with the ASP.NET Core framework

    • Understanding of how to create and run ASP.NET Core Web APIs

  • Fundamentals of Web API Development

    • Basic concepts of RESTful APIs

    • Knowledge of HTTP methods (GET, POST, PUT, DELETE)

    • Understanding of JSON data format

  • C# Programming Skills

    • Basic to intermediate knowledge of C#

    • Experience with object-oriented programming concepts

  • Development Environment Setup

    • Visual Studio or Visual Studio Code installed

    • .NET SDK installed (latest version recommended)

  • Basic Understanding of Logging

    • Familiarity with basic logging concepts

    • Understanding of different log levels (e.g., Information, Warning, Error)

  • Application Insights

    • Basic knowledge of what Application Insights is and how it works

    • Familiarity with Azure services (optional but helpful)

  • Serilog

    • Basic understanding of logging libraries in .NET

    • Knowledge of how to add and configure NuGet packages

  • ELK Stack Overview

    • Basic understanding of Elasticsearch, Logstash, and Kibana

    • Awareness of how these tools work together for log management and analysis

  • Tools Installation

    • Knowledge of how to install and set up tools like Elasticsearch, Logstash, and Kibana if using the ELK Stack locally

    • Basic understanding of Azure portal for Application Insights setup

  • Version Control Systems (Optional but Recommended)

    • Familiarity with Git for source control management

    • Experience with committing and managing code changes

Table of Contents

  • Introduction to Monitoring and Logging

  • Setting up Application Insights

  • Configuring Logging with Serilog

  • Analyzing Logs and Metrics

  • Best Practices for Monitoring and Logging

  • Conclusion

Introduction to Monitoring and Logging

Why Monitoring and Logging Matter

When you build web APIs, it's important to keep an eye on how they are performing and to understand what's happening behind the scenes. This is where monitoring and logging come in.

  • Monitoring helps you track the health and performance of your API. It tells you if everything is running smoothly or if there are any issues, like slow responses or errors.

  • Logging records detailed information about the requests your API handles and any problems it encounters. This helps you troubleshoot issues and understand how your API is being used.

Overview of Application Insights and ELK Stack

To effectively monitor and log your ASP.NET Core Web APIs, you can use powerful tools like Application Insights and the ELK Stack.

  • Application Insights:
    This is a tool from Microsoft Azure that helps you monitor the performance and usage of your web APIs. It collects data about how your API is running and provides insights into any issues. You can use it to track response times, errors, and more.

  • ELK Stack:
    This is a collection of three tools—Elasticsearch, Logstash, and Kibana. Together, they help you collect, search, and visualize your log data.

    • Elasticsearch stores and indexes your log data so you can quickly search through it.

    • Logstash helps you collect and process log data from different sources.

    • Kibana provides a user-friendly interface to visualize your log data and generate reports.

Setting Up Application Insights

Creating and Configuring an Application Insights Resource

  1. Sign in to Azure Portal:

    • Go to Azure Portal and sign in with your Microsoft account.

  2. Create a New Application Insights Resource:

    • In the Azure Portal, click on "Create a resource" in the top left corner.

    • Search for "Application Insights" and select it.

    • Click "Create" and fill in the required information:

      • Name: Give your Application Insights resource a name.

      • Subscription: Choose your Azure subscription.

      • Resource Group: Select an existing resource group or create a new one.

      • Region: Choose the region closest to where your application is hosted.

    • Click "Review + Create," then "Create" to finish the setup.

  3. Get the Instrumentation Key:

    • Once the resource is created, go to your Application Insights resource in the Azure Portal.

    • Find the Instrumentation Key under "Overview" or "Properties." You'll need this key to integrate with your ASP.NET Core project.

Integrating Application Insights with Your ASP.NET Core Project

  1. Install Application Insights SDK:

    • Open your ASP.NET Core project in Visual Studio or Visual Studio Code.

    • Open the NuGet Package Manager and search for Microsoft.ApplicationInsights.AspNetCore.

    • Install the package by running:

        dotnet add package Microsoft.ApplicationInsights.AspNetCore

  2. Add Application Insights to Your Project:

    • Open the Startup.cs file in your project.

    • In the ConfigureServices method, add the following code to configure Application Insights:

        public void ConfigureServices(IServiceCollection services)
  {
      services.AddApplicationInsightsTelemetry("YOUR_INSTRUMENTATION_KEY_HERE");
      services.AddControllers();
  }

    • Replace "YOUR_INSTRUMENTATION_KEY_HERE" with the Instrumentation Key you obtained earlier.

  3. Configure Basic Monitoring:

    • Application Insights will automatically track request rates, response times, and failure rates.

    • To add custom telemetry (e.g., tracking specific events or exceptions), use the TelemetryClient class:

        public class HomeController : Controller
  {
      private readonly TelemetryClient _telemetryClient;

      public HomeController(TelemetryClient telemetryClient)
      {
          _telemetryClient = telemetryClient;
      }

      public IActionResult Index()
      {
          _telemetryClient.TrackEvent("Visited Home Page");
          return View();
      }
  }

Setting Up Basic Monitoring for Performance and Usage

  1. Check Performance Metrics:

    • In the Azure Portal, navigate to your Application Insights resource.

    • Go to the "Performance" section to see metrics like response times and request rates.

  2. Monitor Usage:

    • In the Azure Portal, explore the "Usage" section to view data on how often your API is called and its performance over time.

  3. View Logs:

    • Application Insights collects logs and metrics. You can explore these in the "Logs" section to analyze and diagnose issues.

By following these steps, you'll set up basic monitoring and gain insights into the performance and usage of your ASP.NET Core Web APIs.

Configuring Logging with Serilog

Introduction to Serilog and Its Benefits

Serilog is a popular logging library for .NET applications that makes it easy to capture and manage logs. Logs are essential for understanding what happens in your application and diagnosing issues. Serilog provides structured logging, meaning you can log data in a more organized way, making it easier to search and analyze.

Benefits of Serilog:

  • Structured Logging:

    Logs are stored in a structured format, which helps in filtering and querying logs.

  • Flexible Configuration:
    You can easily configure where and how your logs are stored.

  • Rich Ecosystem:
    Supports various log sinks (places where logs are stored), such as files, databases, or the console.

Installing and Configuring Serilog in Your ASP.NET Core Project

  • Add Serilog to Your Project:

    • Open your project in Visual Studio or Visual Studio Code.

    • Use NuGet Package Manager to install the following packages:

      • Serilog.AspNetCore

      • Serilog.Sinks.Console (for logging to the console)

      • Serilog.Sinks.File (for logging to a file)

You can install these packages using the NuGet Package Manager Console with these commands:

        dotnet add package Serilog.AspNetCore
        dotnet add package Serilog.Sinks.Console
        dotnet add package Serilog.Sinks.File

  • Configure Serilog in Program.cs:

    • Open Program.cs in your ASP.NET Core project.

    • Add Serilog configuration before building and running the host.

      Here’s a simple example of how to set up Serilog:

        using Serilog;

  public class Program
  {
      public static void Main(string[] args)
      {
          // Set up Serilog
          Log.Logger = new LoggerConfiguration()
              .WriteTo.Console() // Log to the console
              .WriteTo.File("logs/myapp.txt", rollingInterval: RollingInterval.Day) // Log to a file
              .CreateLogger();

          try
          {
              Log.Information("Starting up");
              CreateHostBuilder(args).Build().Run();
          }
          catch (Exception ex)
          {
              Log.Fatal(ex, "Application startup failed");
          }
          finally
          {
              Log.CloseAndFlush();
          }
      }

      public static IHostBuilder CreateHostBuilder(string[] args) =>
          Host.CreateDefaultBuilder(args)
              .UseSerilog() // Use Serilog for logging
              .ConfigureWebHostDefaults(webBuilder =>
              {
                  webBuilder.UseStartup<Startup>();
              });
  }

Setting Up Different Log Sinks

Serilog allows you to log messages to different locations, known as sinks. Here’s how to set up some common ones:

  • Console Sink:

    • This sink outputs logs to the console, which is useful during development.

    • Configure it by adding .WriteTo.Console() in your Serilog configuration.

  • File Sink:

    • This sink writes logs to a file, which is helpful for long-term storage and later analysis.

    • Configure it by adding .WriteTo.File("path/to/logfile.txt") in your Serilog configuration.

    • You can also specify options like rolling intervals to create new log files daily or hourly.

  • Other Sinks:

    • Serilog supports many other sinks like databases, Seq, and more. You can explore these options in the Serilog documentation.

Analyzing Logs and Metrics

Understanding and using logs and metrics is crucial for maintaining and improving your ASP.NET Core Web APIs. Here's a guide to follow:

Accessing and Interpreting Application Insights Metrics

  • What It Is: Application Insights provides a set of metrics and data about your application's performance, usage, and errors.

  • How to Access:

    1. Open Azure Portal: Go to the Azure portal and find your Application Insights resource.

    2. Navigate to Metrics: Select the “Metrics” option from the menu.

    3. Explore Metrics: Use the various charts and graphs to view metrics like request rates, response times, failure rates, and more.

  • What to Look For:

    • Performance Metrics: Check response times and request rates to ensure your API is performing well.

    • Failure Rates: Look for any spikes in failures or errors to identify potential issues.

Analyzing Serilog Logs to Track Errors and Debug Issues

  • What It Is: Serilog is a logging library that helps you track and record information about your application’s operations, including errors and important events.

  • How to Analyze:

    1. Locate Logs: Find where Serilog is configured to store logs (e.g., text files, databases, etc.).

    2. Search and Filter: Use search and filtering options to find specific log entries related to errors or warnings.

    3. Understand Log Entries: Read through the logs to understand what errors occurred and trace back to the cause.

  • What to Look For:

    • Error Logs: Identify error messages and stack traces to debug issues.

    • Warning Logs: Look for warnings that might indicate potential problems.

Using ELK Stack for Advanced Log Analysis

  • What It Is:
    The ELK Stack (Elasticsearch, Logstash, Kibana) is a powerful set of tools for collecting, analyzing, and visualizing logs.

  • How to Use:

    1. Logstash:
      Configure Logstash to collect and process logs from various sources.

    2. Elasticsearch:
      Use Elasticsearch to index and store your log data for easy searching and querying.

    3. Kibana:
      Use Kibana to create visualizations and dashboards to analyze your logs and metrics.

  • What to Look For:

    • Visualizations:
      Create charts and graphs to identify trends and patterns in your log data.

    • Search Queries:
      Use advanced search queries to dig deeper into specific issues or events.

Best Practices for Monitoring and Logging

Tips for Effective Logging and Monitoring

  • Log Meaningful Information:
    Capture important details like error messages, request data, and user actions. This helps you quickly understand what’s happening in your application and diagnose issues.

  • Use Different Log Levels:
    Categorize logs by severity—Information, Warning, Error. This makes filtering and finding the necessary logs easier based on the situation.

  • Set Up Alerts:
    Configure alerts for critical errors or performance issues. This way, you get notified immediately if something goes wrong, allowing you to address problems quickly.

  • Monitor Performance Metrics:
    Track key performance indicators (KPIs) like response times and resource usage. This helps you keep your API running smoothly and detect issues before they impact users.

Common Pitfalls to Avoid

  • Logging Too Much Data:
    Avoid overwhelming your log files with too much information. Focus on capturing useful data to prevent log bloat and make it easier to find relevant information.

  • Ignoring Log Rotation:
    Set up log rotation to manage the size of your log files. This prevents your logs from consuming too much disk space and keeps your logging system efficient.

  • Neglecting Security:
    Be cautious with sensitive information. Avoid logging personal data or credentials that could pose a security risk if exposed.

  • Overlooking Log Management:
    Without a proper log management strategy, it can be challenging to search through logs and identify issues. Use tools and techniques to organize and analyze logs effectively.

Recommendations for Maintaining and Scaling Your Logging Setup

  • Regularly Review Logs:
    Periodically check your logs to ensure they’re providing the information you need. Adjust logging settings based on evolving requirements or issues.

  • Scale with Your Application:
    As your application grows, make sure your logging setup can handle increased data. Consider using scalable solutions like cloud-based logging services or distributed log management systems.

  • Automate Log Analysis:
    Use tools that automate log analysis and generate reports. This saves time and helps you quickly identify patterns or recurring issues.

  • Document Your Logging Strategy:
    Keep a record of your logging and monitoring practices. This ensures that everyone on your team follows the same procedures and understands how to use the logging system effectively.

Conclusion

Recap of Key Points

In this guide, we explored how to effectively monitor and log your ASP.NET Core Web APIs. Here’s a quick recap of what we covered:

  • Setting Up Application Insights:
    We learned how to configure Application Insights to track the performance and usage of your APIs, helping you spot any issues or areas for improvement.

  • Configuring Logging with Serilog:
    We set up Serilog to handle error tracking and debugging, making it easier to diagnose problems and keep track of what’s happening in your application.

  • Analyzing Logs and Metrics:
    We discussed how to analyze the logs and metrics collected by Application Insights and Serilog to identify and address performance bottlenecks.

Additional Resources for Further Learning

To continue expanding your knowledge, check out these resources:

  • Microsoft Documentation on Application Insights:
    Learn more about how to use and configure Application Insights here.

  • Serilog Documentation:
    Explore Serilog’s features and how to integrate it into your projects here.

  • ELK Stack Documentation:
    Explore Elasticsearch, Logstash, and Kibana with the official ELK Stack documentation.

  • Online Tutorials:
    Platforms like Pluralsight, Udemy, and Codecademy offer courses on ASP.NET Core, logging, and monitoring.

  • Community Forums:
    Join discussions on Stack Overflow and GitHub to ask questions and share your experiences.

Thank you for following along with this guide. I hope you found it helpful and are now ready to implement effective monitoring and logging in your ASP.NET Core Web APIs. Stay tuned for the next article in the "Mastering C#" series: Securing Third-Party Integrations in ASP.NET Core Web APIs: OAuth, OpenID Connect, and API Keys

Happy coding!

20
Subscribe to my newsletter

Read articles from Opaluwa Emidowo-ojo directly inside your inbox. Subscribe to the newsletter, and don't miss out.

elk-stackloggingdotnetbackendsoftware developmentTechnical writing Web API

Written by

Opaluwa Emidowo-ojo
Opaluwa Emidowo-ojo