My Journey with BullMQ: Sending Bulk Emails

Yesterday, I was working on my personal project, a web app that needed to send bulk emails to users. Everything was going smoothly until I hit a roadblock. My app was starting to slow down because it was trying to handle too many email-sending tasks at once. I needed a way to manage these tasks in the background without overloading my server.

After some research, I discovered BullMQ, a powerful library for managing queues in Node.js applications. It sounded like the perfect solution to my problem, so I decided to give it a try.

Getting Started

First things first, I needed to install BullMQ. I opened my terminal and ran:

npm install bullmq

Since BullMQ relies on Redis for its operations, I made sure Redis was installed and running on my machine.

Creating My First Queue

With the installation done, it was time to create my first queue for sending bulk emails. I created a new file called queue.js and started coding. I imported the necessary modules from BullMQ and set up my queue:

const { Queue } = require('bullmq');

// Create a new queue
const emailQueue = new Queue('email-queue');

// Adding an email task to the queue
emailQueue.add('send-email', { to: 'user@example.com', subject: 'Hello!', body: 'This is a test email.' });

console.log('Email task added to the queue');

I ran the file with node queue.js, and just like that, my first email task was added to the queue. It felt great to see my code working!

Processing Email Tasks

Next, I needed a worker to process the email tasks in the queue. I created another file called worker.js and wrote the following code:

const { Worker } = require('bullmq');
const nodemailer = require('nodemailer');

// Configure the email transporter
const transporter = nodemailer.createTransport({
    service: 'gmail',
    auth: {
        user: 'your-email@gmail.com',
        pass: 'your-email-password'
    }
});

// Create a worker to process email tasks
const worker = new Worker('email-queue', async job => {
    const { to, subject, body } = job.data;

    let mailOptions = {
        from: 'your-email@gmail.com',
        to,
        subject,
        text: body
    };

    try {
        await transporter.sendMail(mailOptions);
        console.log(`Email sent to ${to}`);
    } catch (error) {
        console.error(`Failed to send email to ${to}: ${error.message}`);
    }
});

worker.on('completed', (job) => {
    console.log(`Email task ${job.id} completed`);
});

worker.on('failed', (job, err) => {
    console.log(`Email task ${job.id} failed with error ${err.message}`);
});

I ran this file using node worker.js, and my worker was ready to process email tasks. When I saw the messages in the console confirming the emails were sent, I felt a sense of accomplishment.

Scheduling Bulk Emails

With the basics working, I decided to explore some of BullMQ's advanced features. I wanted to schedule bulk emails to be sent at a later time, so I modified my queue.js to add a job with a delay:

emailQueue.add('send-email', 
    { to: 'user@example.com', subject: 'Hello!', body: 'This is a scheduled email.' }, 
    { delay: 5000 } // Delays the email by 5 seconds
);

I also wanted to track the progress of sending emails and listen for events. I updated my worker.js:

worker.on('progress', job => {
    console.log(`Email task ${job.id} is in progress`);
});

worker.on('completed', job => {
    console.log(`Email task ${job.id} completed`);
});

worker.on('failed', (job, err) => {
    console.log(`Email task ${job.id} failed with error ${err.message}`);
});

Conclusion

By the end of the day, I had integrated BullMQ into my project, and it was handling the bulk email tasks efficiently. My web app was no longer slowing down, and I felt more confident in my project's scalability. BullMQ turned out to be a lifesaver, providing a robust solution to my problem with its easy-to-use API and powerful features.

So, if you ever find yourself needing to manage background tasks, like sending bulk emails, in your Node.js applications, give BullMQ a try.