Agenda 101: Improving communication between microservices

Side Talk

There has been so much debate on the best way to send data within microservice applications. Perhaps, you've also had long discussion within your team on this problem.

Various methods and technique are available ranging from the old http request to kafka. These methods have their own pros and cons and you must way it carefully within your system to determine which one is the best.

However, Agenda provides a simple yet powerful method to process data within your microservice architecture.

Let's dive in to see what agenda is and how to use agenda.

---

What is Agenda?

Agenda is an open-source, flexible job scheduling library designed for Node.js environments. It helps developers manage task scheduling with the support of MongoDB as its persistence layer. Jobs can be scheduled to run periodically, on-demand, or after specific time intervals. This can also be used to as a communication method between microservice architecture.

Key features include:

• Cron-like scheduling.

• MongoDB-backed persistence.

• Priority and concurrency management.

• Repeating jobs and event-backed queues.

Why Use Agenda?

Agenda provides several advantages for job scheduling in Node.js environments:

• MongoDB-based Persistence: Agenda is ideal for apps already using MongoDB, allowing jobs to be stored persistently without requiring additional infrastructure (like Redis).

• Cron and Human-Readable Scheduling: It supports cron expressions as well as readable formats like "every 3 minutes," simplifying scheduling for users who prefer not to use cron syntax.

• Concurrency ande Priority Control: You can manage how many jobs run simultaneously and assign priorities to different jobs.

• Event-Based System: You can hook into different job events (such as completion or failure) to perform additional operations.

• Flexibility in Job Definitions: The library allows you to define jobs with varying configurations, including specifying lock times and concurrency levels.

Implementation (with code examples)

Setting up Agenda is straightforward. Here's an example that walks through the basic steps of configuring and scheduling jobs using Agenda:

Install Agenda and MongoDB in your project.

npm install @hokify/agenda mongodb

Basic Setup: Initialize Agenda by connecting it to a MongoDB instance:

const Agenda = require('agenda');

const mongoConnectionString = 'mongodb://127.0.0.1/agenda';

const agenda = new Agenda({ db: { address: mongoConnectionString } });

Defining Jobs: Define jobs by specifying a task that needs to be run:

agenda.define('delete old users', async (job) => {
    // Your task logic here
    console.log('Deleting users inactive for 2 days...');
});

Scheduling Jobs: You can schedule jobs to run at regular intervals using every() or at specific times using schedule(). For example:

(async function() {
    await agenda.start(); // Start the job scheduler
    await agenda.every('3 minutes', 'delete old users'); // Run every 3 minutes
    await agenda.schedule('in 20 minutes', 'send email report', { to: 'user@gmail.com' });
})();

Handling Concurrency: You can configure job concurrency by setting maxConcurrency and defaultConcurrency options.

agenda.define('send email report', async (job) => {
    const { to } = job.attrs.data;
    console.log(`Sending email to ${to}`);
}, { priority: 'high', concurrency: 5 });

Event-Based implementation in microservice

In a microservice architecture, Agenda can be used to manage scheduled background tasks across services. Since microservices often communicate via APIs, queues, or events, Agenda can be implemented in a service dedicated to job scheduling and execution, while other microservices handle specific functionalities.

Example:

1. Email Microservice: Handles sending emails.

2. Notification Microservice: Schedules jobs with Agenda.

// Notification Microservice - Schedule a job
const agenda = new Agenda({ db: { address: 'mongodb://localhost/agenda' } });

agenda.define('send weekly report', async (job) => {
    const { userId } = job.attrs.data;
    // Make an API call to the Email Microservice
    await axios.post('http://email-service/send', { userId });
});

agenda.every('1 week', 'send weekly report', { userId: 123 });

3. Email Microservice: Receives the job execution request via API.

// Email Microservice - Send email
app.post('/send', async (req, res) => {
    const { userId } = req.body;
    const user = await getUserById(userId);
    emailService.send({
        to: user.email,
        subject: 'Your Weekly Report',
        text: 'Here is your report...'
    });
    res.sendStatus(200);
});

In this setup:

• Agenda runs in the notification microservice, scheduling and managing tasks like sending weekly reports.

• Email microservice performs the actual email-sending task triggered by the job. This approach maintains separation of concerns, with each service handling its specific task.

Additional services (e.g., payment, data processing) can be integrated in a similar manner, where Agenda schedules jobs, and the respective service completes them.

Conclusion

Agenda is a powerful and flexible job scheduling solution for Node.js applications, and MongoDB persistence can be easily added to your stack. Its ability to handle cron-like job scheduling, job concurrency, priority management, and persistent storage makes it highly suitable for a wide range of use cases, from simple task automation to complex background job management. Whether you're building a large-scale SaaS application or a personal project, Agenda provides the necessary tools to effectively manage and execute jobs in the background, ensuring seamless task automation and better resource management.

Its simplicity in implementation, combined with robust features such as concurrency control and event-driven job handling, makes Agenda a great choice for developers looking for a reliable job scheduling solution.