Recap

In Part 2 of this series, I built on foundational concepts and dove into a practical implementation of a simple account management system, using both a conventional CRUD approach as well as an Event Sourcing approach.

In this post, I will expand on the initial implementation of the event-sourced account management system. I aim to guide developers by highlighting best practices I have found and common mistakes to avoid so that they may increase the robustness and maintainability of their systems. You can find the branch for this post in my GitHub repository.

Avoid Common Mistakes With Clear Event Types And Structures

In any event-driven architecture, the most critical pieces of information are the events. I began the design of the account management system with the following interface and base event type:

public interface IAccountEvent
{
    public uint Id { get; set; }
    public Guid AccountId { get; init; }
    public uint Version { get; }
    public DateTime EventDate { get; }
}

public record BaseEvent(Guid AccountId, uint Version) : IAccountEvent
{
    public uint Id { get; set; }
    public DateTime EventDate { get; set; }
}

I have clearly defined the expected properties that all events should contain through the IAccountEvent interface and enforced how these properties are set through the BaseEvent implementation. Given this initial implementation, let us discuss several best practices for maintaining and extending existing event types and structures.

Be Specific

Use precise and descriptive event names to increase clarity. Specificity will help consumers understand what happened without ambiguity.

Don't use generic event names.

public record AccountEvent(Guid AccountId, uint Version) : BaseEvent(AccountId, Version);

This event is too generic. What happened?

Instead, use concise event names with clear intentions.

public record ActivateAccountEvent(Guid AccountId, uint Version) : BaseEvent(AccountId, Version);
public record DeactivateAccountEvent(Guid AccountId, uint Version) : BaseEvent(AccountId, Version);

Make Events Self-Sufficient

Don't add too little.

public record AccountStatusChangedEvent(Guid AccountId, uint Version, string Status) 
    : BaseEvent(AccountId, Version);

What was the previous status? "Status" is not distinctive enough.

Instead, use explicit events with only the required information for better clarity.

public record AccountFrozenEvent(Guid AccountId, uint Version, string Reason) : BaseEvent(AccountId, Version);

Keep It Simple Stupid

Each event should contain strictly necessary data related to the action. Simplicity will reduce excessive or unnecessary data storage and make event processing more efficient.

Don't use nullable or redundant properties.

public record AccountUpdatedEvent(Guid AccountId, uint Version, string? Email, string? Password, string? Address)
    : BaseEvent(AccountId, Version);

This event contains unnecessary data, which increases the bloat in your event storage. Additionally, the consumer is not immediately aware of what was updated. Ambiguity increases the computational load when replaying the event (extra logic is needed to identify what was updated).

Instead, use specific events with required properties.

public record AccountEmailUpdatedEvent(Guid AccountId, uint Version, string OldEmail, string NewEmail) 
    : BaseEvent(AccountId, Version);

Enforce Data Integrity

The initial BaseEvent has public setters for Id and EventDate.

public record BaseEvent(Guid AccountId, uint Version) : IAccountEvent
{
    public uint Id { get; set; }
    public DateTime EventDate { get; set; }
}

These properties are set within the persistence layer of the project so I can explicitly enforce their internal setting through the class design.

public record BaseEvent(Guid AccountId, uint Version) : IAccountEvent
{
    public uint Id { get; internal set; }
    public DateTime EventDate { get; internal set; }
}

Clarity of event types and structures creates a robust foundation for an event-driven system, improving its efficiency and resilience. Additionally, you gain the following benefits:

Consistency - Using a standardised event format prevents ambiguity and makes processing predictable through required properties.
Reproducibility - Well-structured events allow accurate state reconstruction and debugging by replaying event history.
Interoperability - Clear schemas make it easier for different services or teams to understand and consume your events for their projections.
Scalability - Properly structured events allow efficient storage, querying, and processing in distributed systems.
Auditability - A well-defined event log with required audit properties provides a reliable audit trail for compliance and debugging.

Build Robust And Reliable Systems

In event-driven architecture, reliable and consistent event processing is crucial for building robust distributed systems. Two fundamental principles that help achieve this are idempotency and immutability. Idempotency ensures that processing the same request or event multiple times produces the same outcome, preventing unintended side effects in case of retries or duplicate messages. Immutable events, on the other hand, are treated as unchangeable facts, enabling traceability, auditability, and easier debugging.

Sense Check

It is crucial to identify which events will significantly impact the aggregate state of the system and handle them with care. Updates with lower risk, such as modifying metadata like descriptions or labels, require less strict idempotency enforcement. It is pragmatic (in these cases) to create events without extensive safeguards, as the potential consequences of duplicating such events are minimal. The focus should instead be on areas where business logic demands higher guarantees, even at the cost of performance, to maintain idempotency.

Important events, such as those that modify account balances by increasing or decreasing funds, require strict idempotency enforcement. It is essential to ensure (in these scenarios) that each unique action will result in creating and applying these events only once. You can enforce idempotency during event creation, application of events, or both, depending on the level of control and consistency needed.

Prevent Duplicate Event Creation

To enforce idempotency at event creation, I added a unique transactionId to an important event:

public record DepositEventV2(Guid AccountId, uint Version, decimal Amount, Guid TransactionId)
    : BaseEvent(AccountId, Version);

Then I created a method (TransactionExists) to check whether a unique transaction exists for a specific account before creating the event:

private bool TransactionExists(Guid transactionId)
{
    return _processedTransactions.Contains(transactionId);
}

Please note: For simplicity, I have added a hash set of transaction IDs to the AccountAggregate:

private readonly HashSet<Guid> _processedTransactions = [];

The idea is to check if a transaction-specific event has already been processed (already exists within _processedTransactions). If the TransactionId is not in the set, apply the event and add the ID to the set. If it is already in the set, ignore the event and take an appropriate action (log a warning if necessary).

You may easily retrofit this in-memory collection with a DB query to find an entry for the specific AccountId with the same TransactionId. This retrofit will only be necessary once the total number of events for a specific AccountAggregate becomes large as there will be a tradeoff in CPU bound work (performance hit of making a DB query) vs memory bound work (performance hit/cost with caching every TransactionId in memory).

Use a unique identifier to avoid duplicate event creation.

public void HandleDepositCommand(DepositCommand depositCommand)
{
    if (!Active)
    {
        throw new ArgumentException("Account is not active.");
    }

    var depositAmount = depositCommand.Amount;

    if (depositAmount <= 0)
    {
        throw new ArgumentException("Amount must be positive.");
    }

    var transactionId = depositCommand.TransactionId;
    var transactionExists = TransactionExists(transactionId);
    if (transactionExists)
    {
        // Transaction has already been processed, throw an exception
        throw new Exception($"Transaction '{transactionId}' has already been processed for Account '{AccountId}'.");
    }

    var depositEvent = new DepositEventV2(AccountId, NextVersion(), depositAmount, transactionId);

    PersistAndApplyEvent(depositEvent);
}

Enforcing idempotency at event creation increases system robustness by allowing the same request to be received any number of times while only taking action once. If I want even stricter control, I can enforce idempotency when applying events.

Apply Events With Care

Use a unique identifier to avoid applying duplicate events.

private void Apply(DepositEventV2 depositEvent)
{
    var transactionId = depositEvent.TransactionId;
    var transactionExists = TransactionExists(transactionId);
    if (transactionExists)
    {
        // Transaction has already been processed, log a warning
        Console.WriteLine($"Transaction '{transactionId}' has already been applied for Account '{AccountId}'.");
    }

    Balance += depositEvent.Amount;
    UpdateAuditProperties(depositEvent);
    _processedTransactions.Add(transactionId);
}

This approach further increases the robustness of the system by preventing the application of duplicate events.

Enforce Concurrency Through Event Versioning

Use a version number for each event.

public record AccountEmailUpdateMetadataAddedEvent(Guid AccountId, uint Version, string Metadata)
    : BaseEvent(AccountId, Version);

Event versioning prevents concurrency mismatches when creating new events. If an event with version 5 already exists for a stream with ID (AccountId) ABC123, a new event with the same version cannot be appended to the stream. It is then up to the developer to decide whether to fail fast and throw an error or to handle the concurrency issue gracefully (which will require additional business logic).

Events Must Be Immutable

Use init for fields or properties that should not change post-creation. Controlling property setters will preserve event integrity and prevent unintended modifications.

Don't allow properties to change after event creation.

public record AccountEmailUpdatedEvent(Guid AccountId, uint Version, string OldEmail, string NewEmail) 
    : BaseEvent(AccountId, Version)
{
    public string? AdditionalInfo { get; set; } // Bad practice!
}

Instead, create a new event and enforce property creation at initialisation.

public record AccountEmailUpdateMetadataAddedEvent(Guid AccountId, uint Version, string Metadata)
    : BaseEvent(AccountId, Version);

Futureproof With Event Versioning And Schema Evolution

Use Schema Versioning for Forward Compatibility

If event structures need modifications, handle them through versioning. Event versioning allows for gradual migration without breaking backwards compatibility. Both DepositEventV2 and DepositEvent can be handled when applying events.

Given an existing event:

public record DepositEvent(Guid AccountId, uint Version) : BaseEvent(AccountId, Version)
{
    public required decimal Amount { get; set; }
}

Don't modify the structure of an existing event.

public record DepositEvent(Guid AccountId, uint Version) : BaseEvent(AccountId, Version)
{
    public required decimal Amount { get; set; }
    public required Guid TransactionId { get; set; }// Bad Practice!
}

Instead, allow the schema to evolve with a new event version.

public record DepositEventV2(Guid AccountId, uint Version, decimal Amount, Guid TransactionId)
    : BaseEvent(AccountId, Version);

The system aggregate can continue to support the application of both events with two separate apply methods. If there is a breaking change between an old and new event, then an event migration may be necessary.

Scalability Improvements

I will most likely dedicate an entire blog to each of the following sections in the future, but I felt it was worth a brief mention of each of them for this blog specifically.

CQRS

Command Query Responsibility Segregation (CQRS) is an architectural pattern that separates read (Query) and write (Command) operations within a system. This separation allows each side to be optimised independently, improving performance, scalability, and security. By tailoring read and write models to their specific needs, CQRS enables more efficient data access, better resource utilisation, and greater flexibility in handling complex business requirements.

Currently, the AccountContext contains logic for both read and write operations (each operation should be abstracted for independent optimisation):

The separation of read and write models enables systems to scale efficiently and optimise for different workloads. However, simply splitting commands and queries is not enough. Read models must remain consistent with the underlying aggregate to provide accurate and performant queries. This is where event projections come in.

Event Projections

Event Projections are an eventually consistent data structure representing the latest version of a specific aggregate. CQRS and event sourcing work together perfectly because event sourcing recommends creating read-only data structures known as projections, which subscribe to events in the system and create eventually consistent models for easy read operations. Each projection is updated when a new event is added to an event stream (aggregate). Reading these projections is efficient as there is no need to reaggregate all the events for the event stream.

By processing and transforming event streams into tailored read models, projections enable CQRS to deliver fast, optimised views of data while ensuring eventual consistency (there may be some delay between an event taking place and this event filtering through to each respective projection).

Materialised Views

As discussed above, implementing projections for event aggregates creates optimised read models. To further enhance database performance, use materialised views (precomputed query results stored in the database) to speed up specific and frequently executed queries. This approach significantly reduces query latency, optimises resource usage, and enables scalable architectures capable of handling high read loads while maintaining consistency with the underlying event store.

Caching

Caching in an event-sourced system can significantly improve scalability by reducing the load on the event store and speeding up read queries, especially for frequently accessed projections or materialised views. By storing precomputed results in memory, caching minimises redundant computations and enhances responsiveness.

However, there is always one quote from Leon Bambrick I remember:

“There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors.“

Caching introduces challenges such as ensuring cache consistency with the event stream, handling cache invalidation efficiently, and managing stale data. If not carefully implemented, it can lead to outdated reads or increased complexity in synchronisation, making it crucial to balance performance gains with data accuracy and maintainability.

Partitioning

Partitioning plays a crucial role in enhancing the scalability and efficiency of event-sourced systems. By distributing events across multiple partitions, systems can process them in parallel, significantly increasing throughput and handling larger event volumes. This approach also minimises contention, allowing different applications or services to access events independently without resource conflicts. Additionally, partitioning improves query performance by limiting searches to relevant partitions, reducing data scans, and speeding up response times. Typically, events are partitioned using a unique identifier (userId or accountId), ensuring all related events remain grouped for efficient processing and retrieval.

Join The Journey

In this series, I aim to take you through learning and implementing the Event Sourcing architectural pattern, sharing the insights and knowledge I have gained (so far).

Each post will highlight key concepts, practical implementations, and real-world applications, providing a comprehensive understanding of Event Sourcing. As I continue to learn and explore this fascinating architectural pattern, I will document my discoveries and experiences, adding value by showing how to overcome challenges and apply best practices.

Whether you are a beginner or looking to deepen your understanding, this series will guide you through the essentials and beyond, making the learning process engaging and informative.

Part 3: Event Sourcing Best Practices

Table of contents