Microkernel architecture in action

What is microkernel architecture

The microkernel architecture style is a flexible and extensible architecture that allows a developer or end user to easily add additional functionality and features to an existing application in the form of extensions, or “plug-ins,” without impacting the core functionality of the system. For more detailed information on this architecture, please refer to this article: https://www.oreilly.com/library/view/software-architecture-patterns/9781098134280/ch04.html

In this article, we will see how this architecture design is implemented in code.

The use case

We will write a program that imports the news automatically and periodically from external sources to our data store. The data sources can vary from RSS Feeds, JSON files or even pure HTML web pages from other websites. The program is a console application that runs periodically every early morning to import the news to a local database, which then be used to display on our website. Here is how we use Microkernel architecture style to design the program:

The components breakdown

Readers/Crawlers - the plugins

These components are responsible for fetching data from external sources. Regardless of the source data format, there will be a corresponding "plugin" that takes the data from the source and parses them into a unified internal data object, known as a Data Transfer Object (DTO). These objects are then passed to the core system, which processes the data before saving it to the internal data store.

To be recognized by the core system, each plugin needs to be registered. In our program, each plugin is a DLL file placed in a folder called "Plugins." The core system loads all DLL files in this folder and identifies all available readers/crawlers during the initialization. Whenever a new "plugin" needs to be added to the system, simply place the DLL into the "Plugins" folder.

Data processor - the core system

The processor is responsible for triggering all registered readers/crawlers to do their work and receive the DTOs from them. These DTOs will go through some validations, duplication check and so on, done by the processor before being persisted to the database.

Implementation

We will implement the Data processor program using C#. To focus on the architectural perspective, the implementation will be kept as simple as possible.

Contracts

The Core system works with the plugins through a few predefined contracts, which are declared in a shared library. Those contracts including the DTO and some interfaces that the plugins will implement up on.

1. NewsData - the DTO

    public struct NewsData
    {
        public required Guid Id { get; set; }
        public required string SourceId { get; set; }
        public required string Title { get; set; }
        public string? Description { get; set; }
        public required string Link { get; set; }
    }
   

2. INewsReaderPlugin - the plugin main interface

   This represents a plugin instance. The core system will interact with plugins through this interface. In this interface, we define a method GetReader(), which is responsible for returning an instance of INewsReader.

    public interface INewsReaderPlugin
    {
        INewsReader GetReader();
    }
   

3. INewsReader

   This interface represents a news reader that will be used by the Core system to trigger the news fetching. It contains a single method FetchNewsAsync which does the actual network traffic to get the news from a specific source, defined by each plugin.

   
    public interface INewsReader
    {
        Task<IEnumerable<NewsData>> FetchNewsAsync(CancellationToken cancellationToken);
    }
   

RSS Feeds Reader - a plugin implementation

We will implement a simple reader that fetches the news from a RSS feed. There are two interfaces need to be implemented, which have been listed above. The implementation detail will be very simple for demonstration. It uses CodeHollow.FeedReader Nuget package to get the feed items from a hard coded url and maps the items to internal NewsData DTO.

The implementation will be placed in a separated library project, which references to the Contracts project.

public class FeedsReader: INewsReader
{
    public async Task<IEnumerable<NewsData>> FetchNewsAsync(CancellationToken cancellationToken)
    {
        var feed = await FeedReader.ReadAsync("https://thanhnien.vn/rss/home.rss");

        return feed.Items.Select(i => new NewsData()
        {
            Id = Guid.NewGuid(),
            SourceId = nameof(FeedsReader),
            Title = i.Title,
            Description = i.Description,
            Link = i.Link
        });
    }
}

And the plugin implementation itself:

public class FeedsReaderPlugin: INewsReaderPlugin
{
    public INewsReader GetReader()
    {
        return new FeedsReader();
    }
}

News processor - the core system

This is a BackgroundService that collects all registered plugins based on the main plugin interface INewsReaderPlugin. Each plugin runs in a separate thread, fetches the news, then puts the results into a queue of NewsData items. Several handlers run in parallel, picking items from the queue and processing them before permanently storing the data in internal storage.

public class NewsProcessor: BackgroundService
{
    private const int MAX_HANDLERS_NUMBER = 5;
    private readonly IEnumerable<INewsReaderPlugin> _plugins;
    private readonly INewsHandler _newsHandler;
    private readonly ILogger<NewsProcessor> _logger;

    public NewsProcessor(IEnumerable<INewsReaderPlugin> plugins, INewsHandler newsHandler, ILogger<NewsProcessor> logger)
    {
        _plugins = plugins;
        _newsHandler = newsHandler;
        _logger = logger;
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            ProcessData(stoppingToken);
        }

        return Task.CompletedTask;
    }

    private void ProcessData(CancellationToken stoppingToken)
    {
        var queue = new BlockingCollection<NewsData>();
        var tasks = Enumerable.Range(0, MAX_HANDLERS_NUMBER)
            .Select((i) => Task.Run(() =>
            {
                _logger.LogInformation($"Start handling from task #{i}");
                HandleNewsInQueue(queue);
                _logger.LogInformation($"Finish handling from task #{i}");
            }, stoppingToken))
            .ToArray();

        Parallel.ForEach(_plugins, (plugin) =>
        {
            var reader = plugin.GetReader();
            var news = reader.FetchNewsAsync(stoppingToken)
                .GetAwaiter()
                .GetResult();

            foreach (var newsItem in news)
            {
                queue.Add(newsItem, stoppingToken);
            }
        });

        queue.CompleteAdding();

        Task.WaitAll(tasks);
    }

    private void HandleNewsInQueue(BlockingCollection<NewsData> newsQueue)
    {
        foreach (var newsData in newsQueue.GetConsumingEnumerable())
        {
            _newsHandler.Handle(newsData);
        }
    }

To wire all of those things and make them work together, there are some code in Program.cs that dynamically load the plugin dlls and register the NewsProcessor as HostedService:

var builder = Host.CreateDefaultBuilder(args);
builder.ConfigureServices((hostContext, services) =>
{
    PluginLoader.LoadPlugins(Directory.GetCurrentDirectory(), services);
    services.AddHostedService<NewsProcessor>();
});

PluginLoader scans the "Plugins" folder to find all implementations of INewsReaderPlugin and registers to ServiceCollection

public static class PluginLoader
{
    private const string PLUGIN_FOLDER = "Plugins";

    public static void LoadPlugins(string folderPath, IServiceCollection serviceCollection)
    {
        var dllFiles = Directory.GetFiles(Path.Combine(folderPath, PLUGIN_FOLDER), "*.dll");

        foreach (var dllFile in dllFiles)
        {
            try
            {
                var assembly = Assembly.LoadFrom(dllFile);
                var types = assembly.GetTypes();

                foreach (var type in types)
                {
                    if (type.IsAssignableFrom(typeof(INewsReaderPlugin)))
                    {
                        serviceCollection.AddScoped(typeof(INewsReaderPlugin), type);
                    }
                }
            }
            catch (Exception exception)
            {
                Console.WriteLine($"Error occured while loading plugin from dll {dllFile}");
            }
        }
    }
}

Full implementation of this small demonstration can be found at https://github.com/TRANCHYTECH/Blog-microkernel-in-action

Conclusion

The microkernel architecture is a simple yet powerful pattern that enables flexibility and extensibility for applications. It leverages Open-closed principle at architectural level. This article provides a concrete example how this architecture style is implemented in a real use case.