How I Used Redis and BullMQ for Background Tasks in My Internship

When I started my internship, I quickly learned that building an app isn't just about making cool buttons and fetching data. Sometimes, your app needs to do a lot of work in the background, like sending a bazillion push notifications. And if you're not careful, your shiny new feature can accidentally turn into a server-crashing nightmare. (Yep, almost happened. Don't worry, it's a rite of passage!)

That's where I met my new best friends for scalable apps: Redis and BullMQ. They helped me solve the "Why is my app so slow when I try to send a simple notification?" puzzle.

The Problem: When Your App Tries to Do Too Much All at Once

Imagine you've just uploaded an awesome video, and all your followers need to know. If our app tried to send a push notification to every single follower (let's say hundreds of thousands!) right then and there, it would be like:

• Your App: "Hold on, I'm trying to send notifications... pant, pant... to... gasp... everyone... system freezes... HELP!"

• Users: "Ugh, why is this app so slow today? Did it crash again?" (Spoiler: it probably did.)

• Me (Intern): Sweating nervously, checking server logs, wondering if I just broke production.

We clearly needed a better way. We needed to tell these long, boring tasks, "Hey, you! Go do your thing in the background. Don't bother the main app!"

The Solution: Our Dynamic Duo – Redis and BullMQ!

So, we decided to build a "task dispatcher" system. Think of it like a super-efficient post office for all the background work. When a new notification needs to go out, we drop it into a special mailbox (our "queue"). Then, dedicated "mail carriers" (our "workers") pick them up and deliver them without interrupting the main show.

Redis: The Flashy, Super-Fast Mailbox!

We chose Redis to be our super-speedy mailbox. Why Redis? Because it's like a lightning-fast memory bank.

• It Never Forgets (Mostly!): Even if our server decides to take a coffee break (aka restart), Redis can remember all the tasks still waiting in the mailbox. No lost notifications!

• "POOF! Task Dispatched!": Adding a task to Redis or pulling one out is so fast, it practically happens before you can blink. Essential for handling tons of tasks.

• Fun Fact: Redis is often called a "data structure store" because it can do more than just simple key-value storage; it handles lists, sets, and more, which makes it perfect for queues!

BullMQ: The Super Organized Postmaster!

While Redis is our speedy mailbox, BullMQ is the brilliant postmaster that keeps everything in order. It's a fantastic library for Node.js that speaks directly to Redis. BullMQ helps us:

• Label the Packages: We can clearly define what each task is (e.g., "Send New Video Notification," "Process Image Upload").

• Assign Mail Carriers: We set up separate "workers" whose only job is to pick up specific types of tasks from the mailboxes.

• Track Every Package: Did the notification get sent? Did it fail? BullMQ knows! No more guessing games.

• "Oops, Try Again!": If a task fails (maybe someone's phone was off), BullMQ can automatically try sending it again later. Phew!

• No Traffic Jams: It makes sure our mail carriers aren't tripping over each other, managing how many tasks they handle at once.

• "Boss, Look!": BullMQ provides tools to see how busy our mailroom is, which tasks are pending, and which ones are causing trouble.

My "Hello World" of Background Tasks: Sending Push Notifications

1. Setting Up Our Notification Mailbox (Queue)

First, we need to tell BullMQ where our notification tasks will wait. It's like putting up a sign for the "Notification Department" mailbox.

JavaScript

// A super simple way to set up our notification waiting list (aka 'queue')
const { Queue } = require("bullmq'); // We're using the BullMQ library
const connection = require('../../config/redis'); // This connects us to our super-fast Redis server!

// Let's create a new mailbox. We'll call it "notificationQueue".
// All notification tasks will line up here.
const notificationQueue = new Queue('notificationQueue', { connection });

module.exports = notificationQueue; // So other parts of our app can use it!

2. Dropping a Task into the Mailbox

When a new video is posted, instead of sending notifications directly, we package up all the notification details and drop them into our notificationQueue. "Here's a job, BullMQ! Handle it when you can!"

JavaScript

// This is the code that adds a new notification task to our special mailbox
const notificationQueue = require('./notificationQueue');

/**
 * Puts a push notification task into the waiting list for our worker to pick up.
 * No more blocking the user experience! Woohoo!
 * @param {Object} options - All the juicy details for our notification
 * @param {String} options.creatorId - The rockstar who created the content
 * @param {String} options.type - What kind of awesome notification is this? (e.g., "new_video")
 * @param {String} options.title - The catchy title of the notification
 * @param {String} options.message - The actual message the user will see
 * @param {Object} [options.data] - Extra bonus data (like video ID, thumbnail image URL, deep links!)
 */
async function sendPushToFollowersByType(options) {
  // We're adding a 'sendPush' task to our queue. BullMQ handles the rest!
  const job = await notificationQueue.add('sendPush', options);
  return job.id; // Here's a unique ID for this specific task, just in case we need to check on it later!
}

module.exports = sendPushToFollowersByType;

3. The Dedicated Worker: Our Notification Delivery Person

Separately, we have a "worker" program running. This guy's sole purpose in life is to constantly check the notificationQueue. When a new task pops up, it grabs it and gets to work sending out those push notifications!

JavaScript

// This is our tireless worker. It never sleeps (unless the server crashes, oops!).
const { Worker } = require('bullmq');
const connection = require('../config/redis'); // Same Redis connection, so it knows where the mailbox is

// For a real app, these would be proper functions interacting with databases and push services:
// const fetchFollowersFromDatabase = require('../models/Follow'); // To find out who follows whom
// const fetchUserPushTokens = require('../models/User'); // To get the actual phone tokens
// const firebaseAdmin = require('../utils/pushNotification/firebaseAdmin'); // Our magical push notification sender

const worker = new Worker('notificationQueue', async (job) => {
  const { creatorId, type, title, message, data = {} } = job.data;

  console.log(`🎉 Worker picking up a job for creator ${creatorId} (Type: ${type}). Let's get these notifications out!`);

  // Step 1: Log this notification event. Because history is cool.
  // Imagine doing something like: await NotificationEvent.create({ creatorId, type, title, message, ...data });

  // Step 2: Find all the awesome followers who should get this notification!
  // This is where we'd hit our database:
  // const followers = await fetchFollowersFromDatabase({ followingId: creatorId, notificationsEnabled: true });
  const followers = ['followerBob', 'followerAlice', 'followerCharlie']; // Placeholder for our imaginary followers

  if (!followers.length) {
    console.log("🦗 Crickets... no followers to notify this time. Moving on!");
    return; // Nothing to do here!
  }

  // Step 3: Get their device tokens! (The unique IDs for their phones)
  // Again, a database query here:
  // const usersWithTokens = await fetchUserPushTokens({ _id: { $in: followers.map(f => f.followerId) } });
  const deviceTokens = ['phoneToken123', 'phoneToken456', 'phoneToken789']; // Imaginary phone tokens!

  if (!deviceTokens.length) {
    console.log("📱 No phone tokens found for these followers. Guess they don't want push notifications!");
    return;
  }

  // Step 4: Time to send the notifications!
  // IMPORTANT: Push services (like Firebase) often have limits, so we send in batches!
  const chunkSize = 500; // Firebase's typical limit
  for (let i = 0; i < deviceTokens.length; i += chunkSize) {
    const chunk = deviceTokens.slice(i, i + chunkSize);
    console.log(`📦 Sending a batch of ${chunk.length} notifications to the push service!`);

    // In a real app, this is where we'd call our actual push notification service:
    // const response = await firebaseAdmin.messaging().sendEach(chunk.map(token => ({
    //   token,
    //   notification: { title, body: message },
    //   data: { ...data, type, creatorId: String(creatorId) }
    // })));

    // Handle any errors from the push service (e.g., token expired)
    // response.responses.forEach((res, idx) => {
    //   if (!res.success) {
    //     console.warn("🚨 Failed for token", chunk[idx], res.error?.message);
    //     // Optional: Mark this token as invalid in your database to avoid future failures
    //   }
    // });
  }

  console.log(`✅ Notification task for creator ${creatorId} completed! Mission accomplished.`);
}, {
  connection, // Our trusty Redis connection again!
});

// Listener for when a task is done. Yay!
worker.on('completed', (job) => {
  console.log(`✨ Job ${job.id} finished successfully. Time for a virtual high-five!`);
});

// Listener for when a task fails. Boo! But we can learn from it.
worker.on('failed', (job, err) => {
  console.error(`💔 Job ${job.id} failed:`, err.message, '...checking the logs for clues!');
});

What I Gained (Besides Not Crashing the Server!)

Using this Redis and BullMQ magic during my internship gave me some seriously cool insights:

• My App Breathed Easier: No more slow app! Users got instant responses while notifications were handled quietly in the background.

• Scalability Superpowers: We could handle way more users and tasks without breaking a sweat. Just add more workers if things get busy!

• "Oops, I Did It Again" Recovery: If a notification failed (it happens!), BullMQ could automatically retry it. So much less manual work for me!

• Happy Developers, Happy Codebase: Splitting up tasks made our code much cleaner and easier to manage.

• Intern Confession: This setup probably saved me from accidentally crashing the production server at least once. Phew!

My internship project with Redis and BullMQ taught me that handling long-running tasks asynchronously is a game-changer for building robust, high-performance applications. If you're looking to make your app fast, reliable, and intern-proof, definitely check these tools out!