Rails Background Jobs - Reconstruct Your Application

Has your Rails application ever experienced a situation where it becomes slow while handling large amounts of data, sending emails, or performing other demanding computations? During those times, you probably wished for a solution that could efficiently handle these time-consuming tasks without negatively affecting the user experience. Enter Rails background jobs - they are a concept that can completely transform your development environment and can wipe out all your worries.

In this article, I will provide you with a thorough understanding of Rails Background Jobs. Whether you're new to the concept and wondering, "What are background jobs?" or looking for guidance on how to effectively implement them in your Rails applications. Let's begin by understanding the need for Background Jobs through a practical example of where it might be needed.

Practical Example

Imagine you are the owner of an e-commerce platform with loads of traffic, where numerous customers are placing orders every minute. Your business must process these orders promptly and keep up with the growing demand. However, without the help of background jobs, your application could struggle to handle the immense workload required to fulfill each order.

Consider there is a surge of orders flooding your platform, and your application needs to perform a series of complex tasks such as calculating prices, checking inventory, generating shipping labels, and sending confirmation emails. These computations would take a considerably long time if they are being implemented in an order or a flow in the same application and server, causing delays and frustrating your customers.

But fear not! This is where Rails' background jobs come to the rescue. By leveraging the power of background jobs, you can assign these resource-intensive tasks to a separate and dedicated workforce. Think of it as an invisible army of workers tirelessly working in the background while your users navigate through your platform, unaware of the heavy computations taking place behind the scenes.

With background jobs, your application can gain the ability to efficiently handle order processing, email sending, and inventory updates without sacrificing the user experience. It ensures that your platform remains responsive and snappy, even during peak activity periods when the workload is overwhelming. Your customers can continue their shopping journey uninterrupted, while the background jobs diligently carry out their assigned tasks.

When to use it?

Now there is a possibility that you might be wondering that, in exactly what scenarios the background jobs should be used. Here are a few scenarios where background jobs can be used efficiently:

1. Performing resource-intensive tasks, such as logging and tracking, by utilizing a queue and background job model, helps alleviate the heavy workload on the database.

2. For operations that involve reading a large amount of data for reporting purposes, even though it may take time to process and aggregate the data, it is more efficient to execute these operations with fewer requests.

3. In situations where there is high service latency caused by factors like slow network connectivity or peak usage periods, it is better to respond to the user with a message acknowledging the delay and requesting their patience. This approach ensures that users are aware of the ongoing processing and allows for serving more users simultaneously.

4. When interacting with external services that are not critical to the immediate operation, such as collecting system history or sending emails, it is advisable to handle these tasks separately. By doing so, the system can remain responsive and efficient while interacting with other sources of information.

5. By organizing independent jobs in a queue, the system becomes more scalable as additional worker nodes can be added to handle the workload effectively. This enables the system to handle increased demands and process jobs concurrently without sacrificing performance. The term used above i.e. Worker nodes are additional computing resources, usually separate services, processes or instances, dedicated to executing background jobs.

   These nodes actively listen to the job queue for incoming tasks and process them one by one. By using multiple worker nodes, the system can parallelize job execution, increasing throughput and reducing processing time.

External Dependencies

There are several different ways to implement background jobs in Ruby on Rails. The most common approach is using a gem such as Sidekiq, Delayed Job, or many more, which provides a simple and efficient way to manage your background jobs. These gems make it easy to enqueue jobs, set priorities, and track their progress.

Sidekiq is a Ruby framework to perform background jobs. It is an open-source job scheduler written in Ruby that is very useful for handling expensive computations and processes that are better served outside of the main web application. Basically, instead of having the main web app thread handle the complex computation, you can instead punt this tedious and memory-consuming task over to your trusty Sidekiq — a background worker that allows you to execute tasks outside of the main web application thread. The image attached shows the high-level architecture of the application would look with Sidekiq & Jobs/Workers:

Sidekiq requires a web server ( Puma, unicorn e.c.t.), Redis server, and an application server to run properly. When creating a Rails application, the web and application servers are usually present already, but you might need to install Redis server manually.

Setting Up Workers in Rails

In Rails, background jobs are termed as either Jobs or Workers. ActiveJob calls it a Job whereas Sidekiq calls it a Worker. You can use either one. Also, note that ActiveJob does not provide access to the full set of Sidekiq options so if you want advanced customization options for your job you might need to make it a Worker.

Unlike Jobs that are provided by default, you will need to set up workers manually. So you can start by creating the workers directory and an application_worker.rb in it.

mkdir app/workers
cd app/workers
touch application_worker.rb

The application_worker.rb will be responsible for defining the base behavior of the workers. Although, this is completely optional and is dependent on your requirements. The file would look something like this:

# app/workers/application_worker.rb

class ApplicationWorker
  include Sidekiq::Worker

  # Add any common configuration or behavior for all workers here

  # Optional method
  def self.before_perform(*args)
    # You can add any shared setup or logic here
  end
end

Once this is set up, this class will behave exactly like the built-in jobs provided by Rails. In the upcoming implementation discussed in the article, we will be using the syntax of Jobs.

Creating Rails Background Jobs

Before starting the creation of a background job you will need to install dependencies like Redis and Sidekiq. If you are not aware of how to do that, this article might help you. Now considering you have Redis and Sidekiq setup on our machine, and have a Rails application setup you can finally start creating and using background jobs. Unlike Services, the background jobs are provided by default, so you don't have to set them up manually.

Let's set up the configurations first. To use Active Job, specify which queueing backend you want to use in your config/application.rb file. For example, if you're using Sidekiq, you would add the following line to your configuration file:

config.active_job.queue_adapter = :sidekiq

Now before actually running a background job, let's start the Sidekiq and Redis servers on different terminals. Use the following commands to start the servers:

redis-server
bundle exec sidekiq

Now consider the following scenario: You have a controller action in your application that is responsible for initiating the sending of bulk emails. To handle this functionality, you have implemented a service that handles the actual process of sending out emails. The controller code would look something like this:

class EmailsController < ApplicationController
  def send_bulk_emails
    # Retrieve email data or recipients from params or database
    emails = params[:emails]

    # Call the email service to send the bulk emails
    BulkEmailService.send_emails(emails)

    # Redirect or render a response
    redirect_to root_path, notice: "Bulk emails sent successfully."
  end
end

The solution mentioned above will work, but it has a potential drawback. Since the processing of bulk emails is done within the core application, it can consume a significant amount of computational power and put a heavy load on the application. As a result, the overall performance of the application may suffer.

To address this issue and improve performance, we can implement a background job mechanism. Instead of processing the bulk emails directly within the controller method, we can offload this task to a background job.

By utilizing a background job, the heavy and complex processing involved in sending bulk emails can be performed in the background, separate from the main application flow. This means that the application can continue serving user requests without being slowed down by the email processing task.

The background job can be called from the controller method, triggering the email-sending process in the background. This allows the application to handle other tasks while the emails are being processed concurrently. As a result, the overall performance of the application will improve, as the email processing is handled separately and does not impact the responsiveness of the application.

Now, to create a job we will use the rails built-in job generator:

rails generate job send_bulk_emails

Once we run this, a file in app/jobs will be created and the content would be somewhat like this:

class SendBulkEmailJob < ApplicationJob
  queue_as :default

  def perform(*args)
    # Do something later
  end
end

Now once this is done let's transfer the logic defined in the controller to the job.

class SendBulkEmailJob < ApplicationJob
  queue_as :default

def perform(emails)
    begin
      BulkEmailService.send_emails(emails)
    rescue StandardError => e
      Rails.logger.error "Error sending bulk emails: #{e.message}"
      raise e
    end
  end
end

And the controller would now look like this:

class EmailsController < ApplicationController
  def send_bulk_emails
    # Retrieve email data or recipients from params or database
    emails = params[:emails]

    # Enqueue the job to send bulk emails
    SendBulkEmailsJob.perform_async(emails)

    # Redirect or render a response
    redirect_to root_path, notice: "Bulk emails sent successfully."
  end
end

Now when this action is called, it will not be performing these tasks on the main application process but will be performing them in the sidekiq server. This will not block or slow down the processing power of the main Rails application. We can use multiple methods like perform_async, perform_now, perform_later e.t.c. to call background jobs.

Types of Rails Background Jobs

Based on use cases, the background jobs can be categorized into two distinct types:

1. One type of job scheduling method is known as Schedule Background Jobs, which combines the concepts of background jobs and cron jobs. In certain scenarios, we may need to implement specific functionalities at regular intervals of time. This is where background jobs become useful.

   To illustrate this, let's consider a situation where we need to send bulk emails to our subscribers every week. Instead of manually triggering this task, we can automate it using background jobs. By incorporating the defined job into a scheduler, we can ensure that it runs automatically at the desired interval.

   To accomplish this, we use a command in the format of a cron expression:

   0 0 * * 0 cd app && bin/rails your_rake_task:trigger_background_jobs

   Instead of writing the jobs directly, we have the option to use a Rails gem. When it comes to dealing with somewhat complicated and universally used tasks like cron jobs, there is a Ruby Gem available that makes programming them much easier and more visually appealing. There are several helpful gems in this regard, but the most commonly used and popular one is called Whenever.

2. Background jobs also serve as a valuable tool for performing basic asynchronous operations in Rails. These jobs are designed to run in response to specific user demands or event listeners. By offloading these tasks to the background, they can be executed independently of the main user interaction with the application, ensuring a smoother and more responsive user experience.

   When a user initiates an action that requires additional processing or time-consuming tasks, such as generating reports, sending notifications, or performing complex calculations, background jobs come into play. Rather than making the user wait for these operations to complete before proceeding, the tasks are delegated to the background, allowing the user to continue using the application without interruption.

   Event listeners also utilize background jobs to handle specific events or triggers within the application. For example, when a new user signs up, an event listener can be configured to perform certain tasks automatically, such as sending a welcome email or updating user statistics. By executing these operations asynchronously in the background, the application remains responsive and can swiftly handle other user requests or events.

Some Useful Tips

1. When using something like Sidekiq, it's recommended to send only necessary information to the job. Inside the job, you can include extra instructions to find records and carry out more complex tasks. Other than that, if you make sure that your job is idempotent and transactional (meaning it can be repeated multiple times without causing issues), it will be even better.

2. Sometimes, you may need to run a job after a specific time, but you don't know the exact time beforehand. In such cases, you can use the "set" keyword to fetch a particular timestamp from the database and schedule the job accordingly. This allows you to rely on a specific timeframe before the job is executed or create a custom time delay using simple Ruby code.

   Here's an example code:

    SomeJob.set(wait: 3.week).perform_later(params)
   

   This code schedules the job called "SomeJob" to be performed later, waiting for three weeks before executing. The job is given some parameters that it will use when it runs.

3. Active Job offers functionality to trigger specific actions at different stages of a job's life cycle. These actions, known as callbacks, can be implemented as regular methods. To register these callbacks, you can use a convenient class method that follows a macro-style approach, similar to other callbacks used in Rails. The callbacks include:

   1. before_enqueue

   2. around_enqueue

   3. after_enqueue

   4. before_perform

   5. around_perform

   6. after_perform

Conclusion

Background Job plays a crucial role in maintaining the efficiency of your application by keeping daily loaded business logic separate from the main application. By adopting this approach, we can significantly reduce complexity within our codebase and make the application's performance much better. We can also create cron jobs in dockerized rails applications, which will be covered in future articles.

So, stay tuned and keep learning.