Have you ever dealt with race conditions in your code?

While working on one of the feature in Leadlly, a platform designed to help students organize and maintain their self-studies, I encountered an interesting issue that got me thinking about race conditions in code.

The app is designed to generate new weekly planners for students every Sunday. Each planner contains multiple topics, and for each topic, a tracker is created to monitor the student's progress. Initially, everything seemed to be working perfectly fine when I was testing the app with just a few users—trackers were being created without any problems, and all operations went smoothly.

But then, as I scaled up the test environment to include a larger number of students and more topics, things started to break down. Trackers were no longer being created as expected, and the system's behavior became inconsistent. Some topic trackers were being generated successfully, while others were not. This inconsistency pointed to a deeper issue.

Diagnosing the Problem: A Race Condition

After digging into the problem, I discovered that multiple trackers were being created simultaneously, and they were all trying to write to the same spot in MongoDB at the same time. This simultaneous access led to conflicts—basically, several processes were attempting to create trackers at the same index, which caused errors and inconsistencies.

This situation is a classic example of a race condition—a situation where multiple processes try to access and modify shared data simultaneously, resulting in unpredictable outcomes.

The Solution: Implementing a Queue System

To solve the issue, I introduced a queue system for the tracker creation process. Instead of allowing all the tracker creation tasks to run at the same time and conflict with each other, I used a queue to process these tasks sequentially.

By integrating BullMQ (a popular message queue library), I was able to manage the tracker creation process in a controlled, orderly fashion. BullMQ allows tasks to be queued up and processed one by one, ensuring that each student's trackers are created without interference from other tasks.

With this system in place, each student's tracker is now generated in sequence, and there's no more chaos or conflict in the database. The result? Smooth and consistent tracker creation across the board.

Learn more about queues and how they work in detail in this blog.

Other Possible Solutions

While the queue system solved my problem efficiently, it's not the only way to handle such race conditions. Here are two other potential approaches:

1. Database Transactions

   One approach to solving race conditions is to use database transactions. By wrapping the operations that create planners and trackers in a transaction, you can ensure that all operations are atomic—meaning either everything is completed successfully, or none of it happens. This ensures that partial or conflicting updates do not occur, as the database guarantees consistency during the entire process.

2. Batch Processing

   Another alternative is to implement batch processing. Rather than creating trackers immediately as requests come in, you could schedule jobs to run in batches at specific intervals (such as every hour or at night). This would allow the system to process tracker creation tasks in bulk, minimizing the risk of race conditions and reducing the load on the database during peak times.

Conclusion

Race conditions can be tricky, especially when scaling an application that handles many concurrent tasks. In Leadlly, the race condition I faced with tracker creation was resolved by introducing a queue system with BullMQ, which ensured that each task was processed sequentially and in an orderly manner. However, solutions like database transactions and batch processing are also viable alternatives, depending on your application's specific needs.

If you're dealing with a similar problem in your application, consider which solution fits best for your use case—whether it's introducing a queue, using transactions, or scheduling batch jobs.

Have you faced similar challenges while scaling your application? How did you handle race conditions?