Debugging BullMQ with PM2: The Hidden Worker Mystery

Introduction

Working with background job processing is a must in modern backend applications. Whether it's sending emails, processing large datasets, or handling scheduled tasks, an efficient queueing system ensures your app stays fast and responsive.

One of the best tools for this is BullMQ—a powerful, Redis-based job queue for Node.js. When paired with PM2, a robust process manager, you get an efficient, scalable setup for handling background jobs.

However, I recently encountered an interesting debugging challenge while using BullMQ with PM2—a scenario where jobs seemed to disappear instantly from the queue. Let’s break down the experience and the key takeaways!

---

Understanding BullMQ and PM2

🔹 BullMQ: The Redis-Powered Job Queue

BullMQ is an advanced job queue built on top of Redis, designed for high-performance background processing. It provides:

• Scalability – Multiple workers can process jobs concurrently.

• Retries & Delays – Automatic job retries and scheduling.

• Priority & Concurrency Control – Fine-tune job execution.

• Event-Driven Architecture – Listen to job lifecycle events.

BullMQ is ideal for task processing like email notifications, report generation, and transactional workflows.

🔹 PM2: The Silent Process Manager

PM2 is a Node.js process manager that ensures applications (or workers) keep running indefinitely. It helps by:

• Automatically restarting crashed processes.

• Logging and monitoring your app’s health.

• Managing multiple instances for scaling.

While PM2 is a great tool, it can sometimes lead to unexpected behavior—especially when working with background jobs.

---

The Debugging Mystery: Disappearing Jobs! 😲

The Setup

I was working on a BullMQ worker to process jobs added by a producer. My worker looked something like this:

import { Worker, Job } from 'bullmq';

export const worker = new Worker("messageQueue", async (job: Job<{ email: string }>) => {
       console.log(`Message rec ID: ${job.id}`);
       console.log("Processing Message");
       console.log(`Sending email to ${job.data.email}`);
}, {
    connection: {
          host: 'localhost',
          port: 6379
    }
});

console.log("You are in the Worker file ============");

worker.on("completed", (job) => {
    console.log(`Job ${job.id} has been processed successfully.`);
});

worker.on("failed", (job, err) => {
    console.error(`Job ${job?.id} failed with error:`, err);
});

worker.on("error", (err) => {
    console.error("Worker error:", err);
});

The Problem

When I ran my producer to add jobs, I noticed they disappeared instantly!

I checked Redis using:

redis-cli
127.0.0.1:6379> LLEN bull:messageQueue:wait

🚨 Output: 0

This meant that no jobs were waiting—but why?

The Discovery

After debugging for a while, I realized that PM2 was already running my worker in the background!

🔍 What was happening?

• PM2 had started my worker process earlier.

• This worker was silently consuming jobs as soon as they were added.

• Since jobs were processed instantly, I never saw them in the queue!

Confirming PM2’s Involvement

To check if PM2 was running a worker, I used:

pm2 list

And there it was—a background worker process named worker that I had forgotten about!

To see real-time logs from this PM2-managed worker:

pm2 logs worker

Sure enough, my jobs were being processed in the background before I could even inspect them in Redis.

---

The Solution: Controlling PM2

Option 1: Stop PM2 When Debugging

If you're debugging, it's best to stop PM2’s worker and run it manually:

pm2 stop worker   # Stop the worker
pm2 delete worker # Remove it completely
node src/worker.ts # Run worker manually

Now, Redis will actually hold jobs in the queue until the worker consumes them.

Option 2: Monitor PM2 Properly

If you want PM2 to manage the worker but still need visibility:

pm2 logs worker  # See real-time logs
pm2 restart worker # Restart worker if needed

Option 3: Use BullMQ UI Tools

To visualize job processing, use tools like:

• Bull-Board – A UI for BullMQ to track job status.

• Arena – Another great UI for Redis-based queues.

---

Key Takeaways

🔹 BullMQ is powerful but needs proper monitoring. 🔹 PM2 automatically manages processes, which can cause silent background execution. 🔹 Always check pm2 list if jobs seem to disappear! 🔹 Use pm2 logs worker to track job execution. 🔹 For debugging, stop PM2’s worker and run it manually. 🔹 Consider a UI tool like Bull-Board for better job visibility.

---

Conclusion

This experience reinforced an important lesson: Great tools are only as good as our understanding of them! If you’re using BullMQ + PM2, make sure you control them before they control you. 😄

Have you faced similar debugging surprises? Let’s discuss in the comments! 🚀

---