In this article, we'll walk through how to build a real-time scalable chap app using socket.io, node js, Redis, Kafka, MongoDB, and react, I'll show you how to build a scalable socket server where in case users increase and join with other socket server they can still communicate with each other. We use Redis pub-sub for communication between servers.

We'll also spin up a message queue Kafka in case DB is down or the number of messages increases data will not be lost the data. The messages from Redis are produced to Kafka and our DB consumes those messages.

Prerequisites

I recommend the following prerequisites before beginning:

Familiarity with JavaScript and the react
Familiarity with node js, socket.io, Kafka, Redis, MongoDB
Spin up Redis, Kafka and MongoDB server on any cloud or docker

Getting started

First, create the project folder and two subfolders for the client and server of the application.

Building our Node.js API server

Next, we're going to be creating the basis of our Node.js API server. The API server's job is to send messages to the client, and process any new messages created in the client. The server will also set up socket.io connection and expose a WebSocket that can be connected to from our React client and also setup Redis.io pub-sub and KafkaJS.

Inside our server directory, we're going to install the dependencies we need to build our API server.

Express.js is a minimalist web framework for Node.js that provides a simple interface for building HTTP listeners as well as other functionality that makes it easy to build web applications in Node.js.
CORS is an NPM package that lets us process cross-origin requests. This is important as we'll be running a separate client and server application (e.g. they will be running on different ports).
Nodemon is a package that will automatically restart our Node.js server after detecting any changes, making local development easy.
Socket.io is a Javascript library that allows us to create real-time, bidirectional communication between web browsers (clients) and a Node.js server.
Redis.io is s robust, performance-focused and full-featured Redis client for Node.js.
KafkaJS is a registered trademark of The Apache Software Foundation and has been licensed for use by KafkaJS.

Now inside the server.js file, we're going to setup our simple Node.js server using Express.js:

Here we're setting up the express js server and spinning up the MongoDB connection using the npm start command to start the server.

Implementing socket.io server and redis pub-sub

Here we spin up the socket.io server and Redis pub-sub.

Now we start listening for the connections from the client and whenever we receive a sendMessage event from the client we publish the message to the Redis MESSAGES channel. Our server is subscribing to the MESSAGES channel and is on a connection with Redis to start listening to the channels whenever the subscriber receives messages from the MESSAGES channel it produces the message to Kafka and emits a message event to the client.

Here messages are published to the Redis.

Publish messages to Kafka and storing in the MongoDB

First, we create the modal for storing the messages in the MongoDB.

For the purpose of not losing data in case messages increase and DB is down, we produce messages to the Kafka message queue that we are receiving from the Redis subscriber and then spin up a consumer that consumes messages from the Kafka producer and stores in the MongoDB.

Here we are producing messages to Kafka and setup a consumer to consume messages and storing in MongoDB. In case of the DB down we pause the consumer for 60 sec so that the DB back in service and start listening to the messages.

Here the messages are produced to the Kafka.

Setting up React Client

Now, it's time to set up the client to create a React app. Here is very simple react application with a single input field.

We are using socket.io-client on the client side for communicating with the server. The code is given below:

here is the real time view of the messaging

Conclusion

This article presents a practical guide to constructing a real-time scalable chat application with Socket.io, Node.js, Redis, Kafka, MongoDB, and React. Leveraging Redis pub-sub for server communication and Kafka as a message queue ensures system resilience and data integrity. The architecture enables seamless scalability, robust communication, and reliable data persistence, essential for handling growing user bases effectively. Here is the source code of this application GitHub.

Building a real-time scalable chat app using Socket.io, node js, Redis, Kafka, MongoDB, and react