Node.js Task Scheduling With Agenda and MongoDB

The ability of an application to be able to set and schedule future tasks is a bonus and sometimes a must have feature for large scalable applications. In Node.js applications, future tasks can be scheduled to be executed using setTimers or clearTimeout. But one disadvantage of this method is that once the application is restarted we tend to lose all memory stored data, making this method not scalable.

A solution to this in Node.js applications is using crons job schedulers to schedule tasks. They are great for executing recurring tasks or saving tasks to run at a later time.

There are multiple cron options available to schedule tasks in Node.js applications eg: node-schedule, node-cron, bull, Agenda, etc.

In this post, we'll be looking at how to use the Agenda library and integrate it into a Node.js application. Agenda is a simple job scheduling library that provides a promise-based API.

Why Agenda

Among the many options available for cron jobs in Node.js applications. What sets the Agenda library apart is its use of MongoDB for persistence, this offers the advantage of less configuration and saves all scheduled tasks in a MongoDB database, It allows you to automate repetitive tasks, perform maintenance tasks, and keep your application running smoothly without requiring manual intervention.

Agenda keeps our code small and scalable.

Setting up Node.Js Server

In this demo, we'll build our backend application using Node.js and the Express framework, creating a mailing service application. We'll schedule simple application logs that can be replaced with complex codes in real-world scenarios.

Create a folder mail-service for our new project. Open the project in your preferred IDE.

Run the code below in the project terminal:

npm init -y

The command above initializes Node.js into our application and generates a package.json file.

After that, install the application's dependencies by running:

npm install express mongoose dotenv agenda

where

• express: is a Node.js framework we can use to create different API endpoints and a lot more.
• mongoose: will enable us to connect and query our MongoDB database easily.
• dotenv: is used to secure and store sensitive data and credentials in our application's environment variables.

Next, we need to create all files and folders needed for this project, we can easily do this by running:

mkdir jobs
touch server.js .env jobs/agenda.js jobs/jobService.js jobs/scheduler.js

The structure of our application should look like this:

Let's start by creating a basic server for our application.

Paste the following code into the server.js file:

const express = require("express");
require("dotenv").config();
const { schedule } = require("./jobs/scheduler");
const app = express();

const port = process.env.PORT || 3000;

app.use(express.json());

app.listen(port, () => {
  console.log(`Server running on port ${port}`);
});

To start the server. Run the following command in the project terminal.

node server.js

The server should be up and listening on port 3000.

In our application, we divided agenda job config into three sections (files):

1. jobService.js: This script contains the method or function that will be executed when the specified or scheduled time is reached.
2. agenda.js: In this file, we will create an instance of Agenda using our MongoDB URL, and define our application agenda jobs.
3. scheduler.js: In this section, we will create the application's agenda scheduler and set the specific time we want our jobs to run using Agenda's schedule methods.

To begin setting up Agenda, we will start by creating our application's job service.

Creating Job Service

In our application, job service refers to the actions or methods that will be executed at the scheduled times. Before configuring our Agenda cron jobs, we must first define the actions to be carried out.

Paste the following code in the job/jobService.js file:

exports.weeklyNewsletter = async name => {
  console.log(`Hello ${name} have a great week`);
  //more project-related code...
};

exports.dailyNewsletter = async () => {
  console.log("Hey there have a Great Day!!!");
  //more project-related code...
};

In the above code, we are creating and exporting two simple methods - weeklyNewsletter and dailyNewsletter - that will be called in the Agenda job to be executed when our scheduled times are met.

These methods return a simple console message.

Now, we can start integrating Agenda into our Node.js application.

Setting up Agenda

To use the Agenda library we have to make use of the MongoDB cluster.

Please refer to this article or video to learn how to make a MongoDB cluster in Atlas (cloud database).

After creating the MongoDB cluster, paste the URI link into the application's .env file:

PORT=3000
MONGO_URI= //Your database URI here

We can now instantiate and configure Agenda to our Node.js application using the MongoDB database.

Copy and paste the following code into the agenda.js file.

const Agenda = require("agenda");
const jobService = require("./jobService");

const dbURL = process.env.Mongo_URI;

// Instatiate Agenda & Connect to Mongo_URI.
const agenda = new Agenda({
  db: {
    address: dbURL,
    collection: "agendaJobs",
    options: { useUnifiedTopology: true },
  },
  processEvery: "1 minute",
  maxConcurrency: 20,
});

// listen for the ready or error event.
agenda
  .on("ready", () => console.log("Agenda started!"))
  .on("error", () => console.log("Agenda connection error!"));

In the above code:

• The Agenda library and jobService.js script were imported.
• Next, we declare a variable to store the URI value for our MongoDB database.
• Instantiate Agenda by providing connection options such as the database URI, collection name, processEvery value, and maxConcurrency value.

processEvery is a configuration option that specifies the frequency at which Agenda will query the database for jobs that need to be processed. It takes in a string interval.

maxConcurrency on the other hand, limits the number of jobs that can run concurrently. It specifies the maximum number of jobs that can be processed at any given moment

Agenda emits events once it is instantiated these events include:

• ready: emitted when Agenda has successfully connected to the database and is ready to process jobs.
• start: emitted when Agenda begins processing jobs.
• stop: emitted when Agenda stops processing jobs.
• error: emitted if an error occurs during Agenda's initialization process.

We can listen to these events and perform actions or log messages based on the event emitted.

The next step still in the agenda.js file is to define Agenda jobs for our application.

Defining Agenda Job

Defining Agenda jobs is the process of specifying the tasks that need to be executed at specific times. In other words, it involves defining the actions that Agenda will perform when the specified time is reached. For instance, we may define a job to send a weekly newsletter to all subscribers every Monday at 9 am.

To create an Agenda job, we'll use the define() method with three arguments: define(jobName, [options], handler). The first argument is the job name, and the second (optional) argument is for job configuration [options]. The third argument is a Promise-based function that serves as the job handler.

Update the agenda.js file. Copy and paste the following code below the existing code in the file:

// previous agenda code

// define all agenda jobs
agenda.define("send-welcome-mail", async job => {
  console.log("Hello Welcome To Our Newsletter");
});

agenda.define("weekly-mail", async job => {
  const { data } = job.attrs;
  jobService.weeklyNewsletter(data);
});

agenda.define(
  "daily-mail",
  {
    priority: "high",
  },
  async job => {
    jobService.dailyNewsletter();
  }
);

agenda.start();

module.exports = agenda;

In the above code, we defined three Agenda jobs using the define() method. And lastly, we called the Agenda start() method. This method is what actually starts the job queue processing. It must be called after we have defined all our Agenda jobs.

This means that when we call agenda.start(), Agenda will begin querying the MongoDB database for any jobs that are scheduled to be run and will execute those jobs accordingly based on their defined schedule.

It's important to note that you should only call agenda.start() once in your application

We can now schedule the defined job to run at a specific time.

Scheduling Agenda Jobs

In Agenda, there are different methods we can use to schedule our defined jobs. One of the ways is by using the schedule() method. However, we can also use other Agenda methods such as:

• now: schedules a named job to run immediately.
• schedule: schedules a named job at a specific time.
• every: repeats the job named at a given interval (e.g., every 15 minutes, weeks, months, etc.);
• cancel: used to clear all saved jobs or clear a specific job by specifying the job's name as its argument.

To learn more about other ways to use the Agenda library, refer to the official documentation.

Update the schedule.js file with the below code to schedule jobs in our application:

const agenda = require("./agenda");

const schedule = {
  sendWelcomeMail: async () => {
    await agenda.now("send-welcome-mail");
  },
  dailyMail: async () => {
    await agenda.every("in 1 minute", "daily-mail");
  },
  weeklyMail: async data => {
    await agenda.schedule("0 0 * * MON", "weekly-mail", data);
  },
  stopMsg: async () => {
    await agenda.cancel({ name: "daily-mail" });
  },
  // .... more methods that schedule tasks at different intervals.
};

module.exports = { schedule };

In the above code, we're using the schedule methods provided by the Agenda library, which takes in the scheduled time as an argument (in minutes, cron, or dates), the job name, and any data we want to pass to the job handler function.

Our Agenda jobs are now scheduled and ready to be used in our application.

Calling Scheduled Jobs with API Endpoints

Next, we will create endpoints that allow us to call and test our scheduled jobs, enabling dynamic task scheduling and execution within our application.

To achieve this, update the server.js file with the following code:

const express = require("express");
require("dotenv").config();
const { schedule } = require("./jobs/scheduler");

const port = process.env.PORT || 3000;

const app = express();
app.use(express.json());

app.get("/", async (req, res) => {
  await schedule.sendWelcomeMail();
  return res.send({
    message: "Welcome To Our Mailing Service",
  });
});

app.get("/daily", async (req, res) => {
  await schedule.dailyMail();
  return res.send({
    message: "Daily mail scheduled successfully",
  });
});

app.post("/weekly", async (req, res) => {
  const { user } = req.body;
  await schedule.weeklyMail(user);
  return res.send({
    message: "Weekly mail scheduled successfully",
  });
});

app.get("/stop", async (req, res) => {
  await schedule.stopMsg();
  return res.send({
    message: "All scheduled mail stopped successfully",
  });
});

app.listen(port, () => {
  console.log(`Server running on port ${port}`);
});

The server.js file now contains routes for our application, when each route is called, our scheduled jobs are executed at the specific times we have set in our code.

Restart the server by running the following command in the application terminal:

node server.js

Now, we can individually test each route to ensure that our application runs smoothly.

Conclusion

we have successfully integrated the Agenda library into our Node.js application. We defined and scheduled jobs using Agenda's methods, and then created endpoints in our server file to call and execute these jobs dynamically. With the Agenda library, we can easily automate recurring tasks and schedule them to run at specific times, making it a valuable addition to our application's functionality.

I'd love to connect with you on Twitter | LinkedIn | GitHub | Portfolio