Event Sourced Data

A common trap

An all too common problem with application design is that over time, we end up with a central data store that far exceeds the original design. If we are using the monolith pattern, this can be even more of a lock-in, as it's common due to its initial flexibility to pick an SQL type of database.

Of course, over time, we will find ourselves with a multi-gigabyte database, even with a small organisation with simple requirements. We may end up with several of these databases, all pulling data from one another, desperately trying to keep in sync at midnight for the next day's work.

We end up with this being the thing holding us back instead of the liberator we initially thought it was. Migrating data takes special tooling and dedicated time. Any changes to this structure can potentially break applications that are connected to it.

That's right, I said applications, plural. One column name changed without thought can bring down your entire platform.

The primary reason we are stuck is the same reason we cannot decide to convert a chocolate cake to a vanilla one. It has already been baked. Every interaction with the data was stored directly in the data.

What is an event store, and why is it better?

So you've got this super complicated database that's been the centre of your business for years? Every change is a chore? Is every update an ordeal? Well, let me tell you, there's a better way! Using an event store with an event ledger can make your life so much easier in a few different ways. Let me break it down for you:

Scale like a boss

If your data store is feeling bloated and unmanageable, an event store can save the day. It's designed to handle lots and lots of data without slowing down or getting too complicated. We'll talk about it later, but it is the simplest possible way to store data. We can scale big.

Flexibility is key

Sometimes you need to change things up, but that can be tough with a monolithic data store. An event store, on the other hand, is super flexible and can handle changes without breaking a sweat. We can pluck events from any point in the timeline, the data you store in those events can be anything.

The brilliance of resilience

Monolithic data stores can be fragile and prone to failure, but an event store is a tough cookie. Even if one part of the system goes down, the rest can keep chugging along without missing a beat. Being a single long table, it is super easy to back up and manage.

Travel Through Time

It can be tough to track changes to your data over time with a monolithic data store. But an event store keeps track of every single thing that happens, so you always know what's going on. We are effectively time travellers. After we take into account all events we can time travel like Marty Mcfly in the Delorean.

Sharing is Caring

If you're using a monolithic data store, it can be tough to share data between different systems. But with an event store, you can reuse your data in all sorts of different ways without a lot of hassle. We don't make reports from the monolithic store, we derive a new set of data using the entire history.

How does it work?

We need to understand the way we store data in an event store, and how this is its superpower.

An event store is a way of keeping track of every little thing that happens in your system. Each event is like a little nugget of information that describes something that's happened, like a customer making a purchase or a new user signing up.

And the cool thing is that each event is stored separately, almost like a little post-it note. So you can easily look at each event and see what's happened, without having to dig through a bunch of other records.

But here's where it gets cool: because each event is stored separately, it means that you can do all sorts of amazing things with your data. You can easily search through your events to find specific things, like all the purchases made by a certain customer. You can use your events to build cool new features and functions for your system, like recommendation engines or detailed custom reports.

And because each event is so simple, it means that you don't have to worry about all the complicated things that can happen with other kinds of databases. You don't have to worry about complex tables or schemas, or about dealing with all sorts of data that you don't need. You can just focus on the events themselves, and use them to build awesome things.

So there you have it! An event store might sound fancy, but it's just a simple way of storing records. And that simplicity is its main benefit because it means that you can do all sorts of amazing things with your data.

In comparison to your current database, which is a complex mess of tables all connected by keys and indexes, an event store is a simple, single table, with everything anybody has ever done, all lined up like cards, with what they did write on each card.

Why is this useful?

It sounds hard to query this. You've lost all connection between the items, and as you can imagine, this single table would be difficult to write a report on. Though not impossible, it would be too much hard work (and resources) to query this table directly. The pattern we are looking for here though is to loop through all records in order.

By looping all the records, we can see a picture emerging of the true state of something in the system. Let's look at an example:

1. A user is created, and their name is 'Bob'

2. 'Bob' updated his name to 'Robert'

3. 'Robert' was deleted

What does our table look like? That's right! It's empty! Robert was deleted, gone, adios!

• 😭 If we were using a regular data store, the most common pattern would be to have a single record representing Robert, and we would not know his rich history.

• 😎 If we kept all the events ever made about Robert, we would know that he previously existed and that he once liked to be called 'Bob'.

So to make the ledger useful, we need to know that it is the single source of truth for all the data. This new single source of truth is so simple that we can easily go to the beginning of time, and loop through everything that ever happened in our system from day zero.

The superpower

Now the killer feature.

If we loop every event, then we can build a database from it. We can use some simple rules to cherry-pick only the events we care about.

For example, we might want to build a report on how many pets each user has. We would need records about the users, and their pets. We would pull every record related to those specific things, and then build another simple table from the events we found.

This would then build a user interface that queries this newly generated data. The data is simpler and easier to navigate because it deals only with the context of pets and users.

Now you're getting it 💥

💡 We don't query the event store, we use it to build custom data sets.

Now that we can build tables, we have a new ability: the data tables we build can be deleted and re-created at any time. We can delete the database and rebuild it. This is particularly powerful for microservices, where there are many smaller databases, one for each service.

🤯 Whoa

So, to recap:

• We do not store data directly in complex databases that are hard to maintain

• If we append every event in the system to an event store then we have the simplest possible history of everything that ever happened

• We build our data sets by walking the event store and cherry-picking the events we care about

• We are now free to delete databases because we know we can rebuild them again

• We query the generated data instead of the event store. The customer using the data has no idea this voodoo is going on behind the scenes.