In my previous article, I explained distributed system and communication between them. In this article, I will cover an asynchronous way to establish communication between services through a method known as messaging using some queuing technologies.

Introduction

Messaging in a broader way is a form of communication between some parties. In distributed system, it refers to the process of sending and receiving messages(data) between some services to provide communication.

Asynchronous Messaging

Asynchronous messaging in distributed system is a pattern where sender and the receiver of a message do not need to communicate directly with each other. The sender sends the message but do not wait for the receiver's response. The sender just continues with its operation without waiting for the receiver's response.

For example, if Service A is a sender and Service B is a receiver, in asynchronous messaging, Service A sends a message to an intermediary system (like a message queue), and Service B retrieves and processes the message from that intermediary. Even though Service A and Service B do not interact directly, the message is still delivered and handled through this intermediary system. Using queuing technologies for messaging in a distributed system effectively manages this asynchronous communication.

Queuing technologies

Queuing technologies are just some set of tools and patterns designed to facilitate asynchronous messaging through message queues and message buses. A message queue serves as an intermediary storage system, holding messages until they are processed. Let’s first explore the message queue and then the message bus, while they share similarities, the message bus offers additional features. There are three general components in a queuing system:

1. Producer: Who produces message.

2. Consumer: Who consumes message.

3. Queue: Place where messages are stored from producer. It store the messages somewhere on the disk until consumer consumes it.

It can be visualized as:

How does a message queue works?

You might have used socials platform like Facebook or Instagram, where user can configure a settings to receive notifications via in-app push notification, email, or SMS when certain events occur such as receiving a comment on your post.

I'm not sure how Facebook have designed it but if I had to design this feature using queuing technology, I would do so: when a user receives a new comment notification, that notification service creates a message (data) and place that in a message queue. Instead of directly sending notifications through email and SMS services in synchronous way, these services would consume messages from the queue. Each service, such as the email service or SMS service, would pull messages from the queue and process them according to its delivery method. This setup decouples the notification generation from the delivery process, allowing for more flexibility and scalability.

So, message queue operates as a middleware between services. There is a message publisher who sends message, a queue where message is stored until they are processed or queue is full, the consumer who consumes and uses the message accordingly.

What problem does Message queue solves?

In previous article, I mentioned a lot of What ifs in synchronous communication like: when calling to some remote service and it is unresponsive, scalability issues where we might increase afferent and efferent coupling between services, etc. These things tend to severely impact overall system's performance and reliability. Message queue addresses these challenges and provides a robust and reliable asynchronous mechanism.

1. Reliability & Inconsistent data: When Service A directly calls Service B and Service B becomes unresponsive, it can lead to failures in both services. This issue is compounded when multiple services are involved; for instance, in a system with 20 or more services, a single unresponsive service can disrupt the entire process and result in inconsistent data.

   Solution: Messaging pattern. Instead of directly calling services, we can rely on some sorta buffer i.e. a message queue. The message will be stored in a queue in a physical queue, until it is processed by some consumer. This buffering mechanism ensures that even if a service is temporarily unavailable, messages are preserved and processed once the service is back online, thereby enhancing reliability and consistency across the system.

2. Reduced in coupling: Message queuing decouples the calling (consumer) and the responder (publisher) service. The consumer processes messages from the queue asynchronously, allowing the services to operate independently of each other. This decoupling improves system flexibility and scalability, as changes to one service (such as updates or failures) have minimal impact on the other.

3. Improved fault tolerance: Message queues enhance fault tolerance by cutting-off failures. If a service encounters issues, messages remain in the queue and can be processed later once the service recovers. Additionally, message queues often include features like retry mechanisms and dead-letter queues to handle persistent issues, ensuring that the system remains robust and messages are eventually processed.

4. Proper error handling (using dead letter queue or poison queue): Most of the message queuing technologies support dead letter queue or poison queue. Imagine a scenario where a publisher sends a message that is faulty or causes an error during processing. Instead of having the system fail or lose the message, it is redirected to a dead letter queue. This queue is specifically designed to capture messages that cannot be processed successfully after several attempts.

Types of Message Queue

There are various types of message queue. Here are some common ones:

1. Point-to-point queue: In a point-to-point queue system, a producer sends a message to a specific queue intended for a single consumer. Once the message is received by the consumer, it processes the message and sends a response back to the producer. If the response indicates success, the message is removed from the queue, confirming that it has been successfully delivered and processed by the consumer. This mechanism ensures that each message is handled by only one consumer, providing reliable message delivery and acknowledgment.

2. Publish-subscribe queue: Here, queue are called "topic". Publisher publishes message to this topic and any number of subscribers subscribed to this topic receives a copy of message. This process is called fanout. Pub-sub queue is ideal when you need to send message to multiple services.

3. In-memory queue: Here, queue stores messages on memory rather than on disk or "database". They are performant but not ideal if you want your data to be persisted and durable.

4. Priority queue: Typically, a message queue operates on a FIFO (first-in, first-out) basis, meaning messages are processed in the order they are received. However, a priority queue allows messages to be stored and processed according to their priority level. In a priority queue, messages with higher priority are dequeued and processed before those with lower priority, regardless of the order in which they were added. This ensures that critical messages are handled quickly, enhancing the responsiveness of the system to high-priority tasks.

Messaging Protocols

Every queuing technology follow a set of rules on how messages are serialized or formatted, how they communicate, and how they are transmitted. Messaging protocols ensures to enable standard among queuing technology. Here are some commonly used messaging protocols:

1. AMQP (Advanced Message Queuing Protocol): AMQP is a widely used robust, language-agnostic, wire-level messaging protocol that facilitates message oriented middleware. Wire-level means it specifies exactly how data should be structured and transmitted over the network. This allows different services to understand each other, no matter what language or platform they are using. This makes AMQP reliable and useful for complex systems that need to exchange messages consistently and securely.

2. MQTT (Message Queuing Telemetry Transport): MQTT is a light weight pub-sub messaging protocol designed for simple and efficient communication, especially over networks that are unreliable. It’s often used in situations where devices have limited processing power or bandwidth, like in IoT applications. MQTT works using a pub-sub pattern, where devices can send messages (publish) to a central broker, and other devices can receive (subscribe to) those messages if they’re interested. This makes it easy to send data from sensors, for example, to a server or other devices, without needing a direct connection between them.

3. STOMP (Simple Text Oriented Messaging Protocol): STOMP is a straightforward messaging protocol that's is easy to understand and work with because it is text-based messages. In other word, they are human readable, making it simple to debug and develop with. STOMP is often used to send messages between different services, like sending updates from a server to a web application. It works by having clients connect to a message broker, where they can send (publish) messages or receive (subscribe to) messages. The text-based nature of STOMP makes it accessible and flexible, allowing it to be used in a variety of applications where simplicity and ease of integration are important.

4. XMPP (Extensible Messaging Presence Protocol): Mainly used for real-time communication, XMPP is a protocol based on XML. XMPP is extensible, meaning you can add custom features to it, like file sharing or voice calls. This flexibility makes it easy to build more complex applications on top of the basic messaging functions. For example, a simple chat app using XMPP can be extended to include video calls, group chats, or even real-time collaboration tools, all while using the same underlying protocol. XMPP supports real-time updates, so you can see if someone is online, away, or busy. This protocol is widely used in chat applications, collaboration tools, and even IoTs, where quick, real-time communication is important.

Understanding some popular queuing choices

There are many popular message queuing technologies out there with its own features and tradeoffs. Here are some popular choices:

1. RabbitMQ: Opensource queuing technology that supports various types of messaging protocols including: AMQP, STOMP, MQTT, HTTP, Web-sockets, etc. It is known for its robustness, and flexibility. RabbitMQ comes with visual interface where you can see and manage queues and exchanges. It is ideal for systems where there are complex routing patterns (meaning there are many rules for deciding where messages should go based on their content), requires data and delivery reliability, and systems using multiple messaging protocols.

2. AWS Simple Queue Service (SQS): A fully managed cloud-based queuing system provided by Amazon Web Services (AWS). Fully managed means AWS takes care of servers and infrastructures. SQS are very scalable. There are two types of queues provided by SQS:

   1. Standard Queue => High throughput and at-least one delivery but doesn't guarantee the order of messages.

   2. FIFO Queue => Messages are processed in order they were sent and only once.

SQS provides options for encrypting messages and controlling access to the queue making it secure and ensuring that only authorized users and services can access or manage the messages. SQS is very ideal for serverless systems.

3. Apache ActiveMQ: Apache ActiveMQ, an open-source message broker that facilitates communication between different systems or components by handling messages supports various messaging protocols like: AMQP, STOMP, and MQTT, making it versatile. Apache ActiveMQ is ideal for JVM applications where versatility, and reliability are must.

Message Bus

A message bus is a more complete messaging framework that serve as a centralized channel for communication between various services in a system. A message bus typically includes all the core features of a message queue, such as reliable message delivery and message storage, and adds additional functionalities for routing, processing, and transforming messages. Like a message queue, it stores messages until they are processed. Additional features like:

1. Advanced routing: The message bus can direct messages to different services based on their content or other factors. For example, it might send customer orders to an order processing service and customer feedback to a support team.

2. Message processing: It can handle and modify messages before sending them out. For example, it might add extra information to a message or filter out unnecessary details.

3. Message Transformation: The message bus can change messages into different formats or structures to fit the needs of different systems. For example, it might convert a message from XML to JSON.

4. Pub-Sub Pattern: Instead of sending messages to one specific place, a message bus can send the same message to multiple destinations. For example, a news update could be sent to both a website and a mobile app (both different services).

5. Orchestration: The message bus manages how messages flow between different services. It ensures that the right messages go to the right places in the correct order. For example, it might ensure that a customer’s order goes through several steps, like payment, shipping, and confirmation (in order).

How does a Message bus works?

Message bus can be described as message queue on roids. While a message queue focuses on storing and delivering messages in a straightforward, point-to-point manner, a message bus takes it up a notch with additional features (mentioned above). Message infrastructure can be a bit complex to understand, but here is a detail look at it:

1. Producer: A service that generate messages. They send data or events to the message bus. For example, a service might generate a message when a user submits a form.

2. Consumer: These are the services that receive and process messages. Consumers subscribe to the message bus to get the messages they are interested in.

3. Message Broker: A component within the message bus that routes messages from producers to the appropriate consumers. It may handle tasks like message transformation, and routing based on various criteria.

4. Message Topics or Channels: These are logical channels or categories through which messages are classified. Producers and consumers can publish or subscribe to specific topics to filter the messages they handle.

5. Message Store: A persistent storage (physical disk) that keeps messages until they are successfully processed. This ensures that messages are not lost in case of failures.

To sum up, a producer generates a message and sends it to the message bus, specifying the intended topic or channel. The message bus, often through its broker, routes the message to the appropriate queue(s) based on the topic or channel specified. The routing may involve applying filters or transformations. The message is temporarily stored in a queue if it cannot be immediately delivered. This ensures that messages are not lost and can be processed later. Consumers subscribe to topics or channels and receive messages from the message bus. The delivery may be direct or involve additional processing based on the bus’s configuration. After processing, the consumer acknowledges receipt of the message. If the processing fails, the message bus might retry delivery or move the message to a dead-letter queue/poison queue for further investigation.

Difference between Message queue and Message bus

What problem does a Message bus solves?

Similar to a message queue, a message bus solves problems related to complex communication patterns, scalability, and system integration. It enhances reliability and fault tolerance by managing message routing, integration across diverse systems, and providing robust monitoring tools. By decoupling services and supporting advanced features like message persistence and transformation, a message bus ensures smooth and scalable communication in complex, distributed environments.

So Message queue or Message bus?

Well, choosing any type of solution really depends upon the requirements.

1. Message Queue: Ideal for simpler use cases where you need reliable, asynchronous communication between components. It focuses on storing and delivering messages between producers and consumers (services), handling scenarios where a producer needs to communicate with a specific consumer.

2. Message Bus: Suited for more complex scenarios involving multiple services and diverse communication patterns. It offers advanced features like message routing, transformation, and integration, making it effective for managing communication in distributed systems with diverse interactions.

In summary, use a message queue for straight forward asynchronous communication, or opt for a message bus when you need a more comprehensive solution with enhanced routing, integration, and scalability capabilities.

Implementation

Now that you’re familiar with messaging and queuing technologies, let’s put them into practice. Consider this scenario: a payment system where a user can send and receive money. When payment is completed, both sender and receiver will receive a promotional email and SMS notification. So there are 3 services, payment.API,promotional.Email and promotional.SMS.

I'll use .NET 8, RabbitMQ, and MassTransit to build this system.

MassTransit is an open-source distributed application framework for .NET that provides a consistent abstraction on top of the supported message transports. The interfaces provided by MassTransit reduce message-based application complexity and allow developers to focus their effort on adding business value.

Basically, masstransit is a wrapper messaging technologies (most of the messaging frameworks and technologies) providing high level abstraction to use any types of messaging or queuing technologies with added functionalities like sagas, retries, transactional, and more to build resilient and robust distributed system.

Here, message publisher is payment.API whereas message receivers are promotional.Email and promotional.SMS.

An exchange is a virtual entity within RabbitMQ that receives messages from producers and routes them to queues based on certain rules. It acts as a central hub for message distribution. There are several types of exchanges in RabbitMQ, but the most common ones are:

1. Direct Exchange: This is the simplest type. Messages are routed to queues based on an exact match between the routing key specified by the producer and the binding key of the queue.

2. Fanout Exchange: This exchange broadcasts messages to all queues.

3. Topic Exchange: This exchange allows for flexible routing based on patterns in the routing key.

4. Headers Exchange: This exchange routes messages based on message headers, not the routing key.

Routing determines how messages are delivered from an exchange to specific queues. Routing Key => A string set by the producer when publishing a message. It's used to determine the destination queue. Binding Key => A pattern associated with a queue. It defines which messages the queue will receive.

By default, MassTransit creates an exchange called amq.default.This is a direct exchange, meaning messages are routed based on an exact match with the routing key.

So, wherever sender transfer fund to a receiver, both of them will get an email and SMS.

payment.API's swagger UI.

1. Seed payment accounts. It will generate random users and other information.

2. View all account.

3. Copy account no and transfer fund.

4. You can check both account in GET request.

Payment transferred from payment.API like so:

This message will be published and processed by other services.

SMS service:

Email service:

This solution offers fault tolerance and resiliency. If either the SMS or email service experiences downtime, messages will be queued and processed when the service becomes available again.

Full source code: Github

Messaging is the solution?

No. It depends. A fundamental principle in distributed system design is to minimize direct communication between services. However, preferring asynchronous communication over sync introduces a tradeoff: request-response.

The decision to implement messaging should be deliberate and based on a careful evaluation of project requirement and trade-offs.

Summary

This article explores asynchronous messaging in distributed systems, focusing on how messaging facilitates communication between services without requiring direct interaction and key concepts like message queues and message buses, explaining their roles in decoupling services, improving reliability, and enhancing fault tolerance. Using RabbitMQ, MassTransit, and .NET 8, I demonstrate building a payment system that sends email and SMS notifications highlighting the benefits of asynchronous messaging, exploring concepts like message queues, message buses, and various protocols (AMQP, MQTT, STOMP). I will try to publish more articles on building data consistency over the system, resiliency, and security next.

References

Messaging: NDC presentation by Chris Patterson

RabbitMQ: docs

MassTransit: Good intro to MassTransit by Milan Jovanović, Nick Chapsas Tutorial

More: About Saga Pattern, Messaging Protocols, Distributed system design fundamental course by Udi Dahan, Async communication in Distributed System

Diagram: draw.io