Mastering Laravel job batching Pt. 1: An overview

Dominique VASSARDDominique VASSARD
21 min read

Table of contents

Introduction

In this series of articles, we'll take a look at Laravel job batches and how to take the most out of it.

This article is 1st of the 4-parts series: Mastering Laravel batches

A bit of context

At AssessFirst, our codebase is mainly focused on operating on one entity at a time.

At some point, we needed to process long lists of operations and wonder what would be the way to go. Laravel jobs and queues are powerful but lacks any kind of control or monitoring that would fit our needs.
We decided to look at Laravel batches and this series of article expose our exploration of them.

Let's get started with an overview of the laravel batches!

Setup the application

First, let's create a new laravel app.
For this, we'll use sail with the specific service mysql as database and redis for queue management:

curl -s https://laravel.build/laravel-batches?with=mysql,redis | bash

Once installed, browse to application directory:

cd laravel-batches

define an alias for sail

alias sail='sh $([ -f sail ] && echo sail || echo vendor/bin/sail)'

Ensure that redis is used as the default queue connection in .env:

QUEUE_CONNECTION=redis

Setup database:

sail artisan migrate

and run the application:

sail up -d

You can check that the default page is correctly displayed but we won't use web anyway.

Meet Laravel batches

As stated in the documentation :

Laravel's job batching feature allows you to easily execute a batch of jobs and then perform some action when the batch of jobs has completed executing.

In our case, batches will allow to run our multiple operations as if they were a single one, with control over failure and with the ability to run a final operation if a specific return is required.

Create a job

Batch are sets of jobs, so let's make our first job:

sail artisan make:job ExampleJob

This job will only write a log with the job's class name and a number:

# app/Jobs/ExampleJob.php

// ...
class ExampleJob implements ShouldQueue
{
    use Batchable, Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    /**
     * Create a new job instance.
     */
    public function __construct(public int $number)
    {
    }

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        Log::info(sprintf('%s [%d] RAN', class_basename($this), $this->number));
    }
}

Notice the use Batchable which will allow to add out job to a batch.

We can run it via tinker and check that the is written.

# Open tinker
sail artisan tinker

# Run the job (synchronously)
> use App\Jobs\ExampleJob
> ExampleJob::dispatchSync(1)
= 0

# Exit tinker
exit

# Check log
tail -n 1 storage/logs/laravel.log 
[2024-05-08 17:09:07] local.INFO: ExampleJob [1] RAN

We're going to tun the batch via a command, so let's create it:

sail artisan make:command --command=batcher Batcher

Or simple batch will run 5 times the ExampleJob.
Logs will hep to understand what's going on:

# app/Console/Commands/Batcher.php

//...
class Batcher extends Command
{
    //...

    /**
     * Execute the console command.
     */
    public function handle()
    {
        Bus::batch(
            Arr::map(
                range(1, 5),
                fn ($number) => new ExampleJob($number)
            )
        )
            ->before(fn (Batch $batch) => Log::info(sprintf('Batch [%s] created.', $batch->id)))
            ->then(fn (Batch $batch) => Log::info(sprintf('Batch [%s] ended.', $batch->id)))
            ->dispatch();
    }
}

Bus::batch(array) initialize the batch with the given job list.
->before() runs when the batch is created but before any job is ran.
->then() runs when batch is finished

To test that our batch works well, it is recommended to open two terminal windows:

  • one will tail the logs
tail -f storage/logs/laravel.log
  • another will run the command:
sail artisan batcher

You should see this in logs:

[...] local.INFO: Batch [9c1b2c1e-d1aa-4787-b774-5a95c1d16db8] created.  
[...] local.INFO: ExampleJob [1] RAN  
[...] local.INFO: ExampleJob [2] RAN  
[...] local.INFO: ExampleJob [3] RAN  
[...] local.INFO: ExampleJob [4] RAN  
[...] local.INFO: ExampleJob [5] RAN  
[...] local.INFO: Batch [9c1b2c1e-d1aa-4787-b774-5a95c1d16db8] ended

If you only see the first log, your queue may not be running.
Open a new terminal window and run this command:

sail artisan queue:work

Don't close this terminal! We'll use it later.

Inspecting a batch

You have 2 solutions to get info about a batch.
First is to check the database: get you batch id and check the job_batches table:

# Connect to database
sail artisan db

# Get the data
mysql> SELECT * FROM laravel.job_batches WHERE id = '9c1b2c1e-d1aa-4787-b774-5a95c1d16db8'\G
*************************** 1. row ***************************
            id: 9c1b2c1e-d1aa-4787-b774-5a95c1d16db8
          name: 
    total_jobs: 5
  pending_jobs: 0
   failed_jobs: 0
failed_job_ids: []
       options: a:2:{s:6:"before";a:1:{i:0;O:47:"Laravel\SerializableClosure\SerializableClosure":1:{s:12:"serializable";O:46:"Laravel\SerializableClosure\Serializers\Signed":2:{s:12:"serializable";s:328:"O:46:"Laravel\SerializableClosure\Serializers\Native":5:{s:3:"use";a:0:{}s:8:"function";s:118:"fn (\Illuminate\Bus\Batch $batch) => \Illuminate\Support\Facades\Log::info(sprintf('Batch [%s] created.', $batch->id))";s:5:"scope";s:28:"App\Console\Commands\Batcher";s:4:"this";N;s:4:"self";s:32:"00000000000002930000000000000000";}";s:4:"hash";s:44:"erZxxohDPSjM8/GJ83u60B7s2ckQH/+/OoWUqqUBYgs=";}}}s:4:"then";a:1:{i:0;O:47:"Laravel\SerializableClosure\SerializableClosure":1:{s:12:"serializable";O:46:"Laravel\SerializableClosure\Serializers\Signed":2:{s:12:"serializable";s:326:"O:46:"Laravel\SerializableClosure\Serializers\Native":5:{s:3:"use";a:0:{}s:8:"function";s:116:"fn (\Illuminate\Bus\Batch $batch) => \Illuminate\Support\Facades\Log::info(sprintf('Batch [%s] ended.', $batch->id))";s:5:"scope";s:28:"App\Console\Commands\Batcher";s:4:"this";N;s:4:"self";s:32:"00000000000002960000000000000000";}";s:4:"hash";s:44:"Avra0qOKksUBzfBFc1DcYeeFvHyUgzClrUQub5/ZsJI=";}}}}
  cancelled_at: NULL
    created_at: 1716405182
   finished_at: 1716405185
1 row in set (0.00 sec)

Or via tinker:

# Open a tinker session
sail artisan tinker

# Get the data
> Bus::findBatch('9c1b2c1e-d1aa-4787-b774-5a95c1d16db8')
= Illuminate\Bus\Batch {#5166
    +id: "9c1b2c1e-d1aa-4787-b774-5a95c1d16db8",
    +name: "",
    +totalJobs: 5,
    +pendingJobs: 0,
    +failedJobs: 0,
    +failedJobIds: [],
    +options: [
      "before" => [
        Laravel\SerializableClosure\SerializableClosure {#5170},
      ],
      "then" => [
        Laravel\SerializableClosure\SerializableClosure {#5168},
      ],
    ],
    +createdAt: Carbon\CarbonImmutable @1716405182 {#5220
      date: 2024-05-22 19:13:02.0 UTC (+00:00),
    },
    +cancelledAt: null,
    +finishedAt: Carbon\CarbonImmutable @1716405185 {#5208
      date: 2024-05-22 19:13:05.0 UTC (+00:00),
    },
  }

We can see the details of jobs:

  • totalJobs is the total num of jobs

  • pendingJobs is the number that still needs to be run.

  • failedJobs is the number of failed jobs

  • options stores all the callback we define at batch creation

Tracking batch progress

Laravel batches offer a nice way to track their progression, we just need to use the progress method, like this:

# app/Console/Commands/Batcher.php

//...
class Batcher extends Command
{
    //...

    /**
     * Execute the console command.
     */
    public function handle()
    {
        Bus::batch(
            Arr::map(
                range(1, 5),
                fn ($number) => new ExampleJob($number)
            )
        )
            ->before(fn (Batch $batch) => Log::info(sprintf('Batch [%s] created.', $batch->id)))
            ->then(fn (Batch $batch) => Log::info(sprintf('Batch [%s] ended.', $batch->id)))
            ->progress(fn (Batch $batch) =>
            Log::info(sprintf(
                'Batch [%s] progress : %d/%d [%d%%]',
                $batch->id,
                $batch->processedJobs(),
                $batch->totalJobs,
                $batch->progress()
            )))
            ->dispatch();
    }
}

The processedJobs() method returns the number of ran jobs, successful or failed.
The progress() method returns a percentage of progression

Let's try it.

sail artisan batcher

If we check the logs, we see this:

[...] local.INFO: Batch [9c1b30b8-3227-4f1f-9730-1af20c496fe3] created.  
[...] local.INFO: ExampleJob [1] RAN  
[...] local.INFO: Batch [9c1b30b8-3227-4f1f-9730-1af20c496fe3] progress : 1/5 [20%]  
[...] local.INFO: ExampleJob [2] RAN  
[...] local.INFO: Batch [9c1b30b8-3227-4f1f-9730-1af20c496fe3] progress : 2/5 [40%]  
[...] local.INFO: ExampleJob [3] RAN  
[...] local.INFO: Batch [9c1b30b8-3227-4f1f-9730-1af20c496fe3] progress : 3/5 [60%]  
[...] local.INFO: ExampleJob [4] RAN  
[...] local.INFO: Batch [9c1b30b8-3227-4f1f-9730-1af20c496fe3] progress : 4/5 [80%]  
[...] local.INFO: ExampleJob [5] RAN  
[...] local.INFO: Batch [9c1b30b8-3227-4f1f-9730-1af20c496fe3] progress : 5/5 [100%]  
[...] local.INFO: Batch [9c1b30b8-3227-4f1f-9730-1af20c496fe3] ended.

Clean and easy, isn't it?

Canceling a batch

What if we want to stop a running batch?
There's also a solution for that.
First, we need to update our ExampleJob in order to have it not doing anything if the batch is cancelled. And in order to be able to cancel it, we'll make it run for a longer time:

# app/Jobs/ExampleJob.php

// ...
class ExampleJob implements ShouldQueue
{
  //...

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        // If batch is cancelled, stop execution
        if ($this->batch()->cancelled()) {
            Log::info(sprintf('%s [%d] CANCELLED: Do nothing', class_basename($this), $this->number));
            return;
        }

        sleep(5);
        Log::info(sprintf('%s [%d] RAN', class_basename($this), $this->number));
    }
}

Next, we need to work on the Batcher command in order to add the possibilty to cancel a job. First, we move the batch execution in its own method

/**
 * Run a batch of ExampleJobs
 *
 * @return void
 */
private function runExampleBatch(): void
{
    Bus::batch(
        Arr::map(
            range(1, 5),
            fn ($number) => new ExampleJob($number)
        )
    )
    //...
}

Then create a method to cancel a batch:

/**
 * Cancel the batch with the given id
 *
 * @param string $batch_id
 * @return void
 */
private function cancelBatch(string $batch_id): void
{
    // Find the batch for the given id
    $batch = Bus::findBatch($batch_id);
    // Cancel the batch
    $batch->cancel();
}

And manage an option tochhose whic method should run.
See the full code of the command below:

# app/Console/Commands/Batcher.php

class Batcher extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'batcher
                            {--cancel= : Cancel batch with the given id}';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Batcher';

    /**
     * Execute the console command.
     */
    public function handle()
    {
        if ($this->option('cancel')) {
            $this->cancelBatch($this->option('cancel'));
        } else {
            $this->runExampleBatch();
        }
    }

    /**
     * Cancel the batch with the given id
     *
     * @param string $batch_id
     * @return void
     */
    private function cancelBatch(string $batch_id): void
    {
        // Find the batch for the given id
        $batch = Bus::findBatch($batch_id);
        // Cancel the batch
        $batch->cancel();
    }

    /**
     * Run a batch of ExampleJobs
     *
     * @return void
     */
    private function runExampleBatch(): void
    {
        Bus::batch(
            Arr::map(
                range(1, 5),
                fn ($number) => new ExampleJob($number)
            )
        )
            ->before(fn (Batch $batch) => Log::info(sprintf('Batch [%s] created.', $batch->id)))
            ->then(fn (Batch $batch) => Log::info(sprintf('Batch [%s] ended.', $batch->id)))
            ->progress(fn (Batch $batch) =>
            Log::info(sprintf(
                'Batch [%s] progress : %d/%d [%d%%]',
                $batch->id,
                $batch->processedJobs(),
                $batch->totalJobs,
                $batch->progress()
            )))
            ->dispatch();
    }
}

Before testing, ensure that code is up to date in queue by restarting them:

sail artisan queue:restart

And run the queue:

sail artisan queue:work

To test, we'll need 3 terminal windows.
On the first one, we'll watch the logs with:

tail -f storage/logs/laravel.log

On the second one, we can launch the batch with:

sail artisan batcher

Once batch is started, grab its id fromp the logs and cancel it using a third window:

sail artisan batcher --cancel 9c1b3bef-7582-4007-973c-ba483306290b

Let's check the logs:

[...] local.INFO: Batch [9c1b3bef-7582-4007-973c-ba483306290b] created.  
[...] local.INFO: ExampleJob [1] RAN  
[...] local.INFO: Batch [9c1b3bef-7582-4007-973c-ba483306290b] progress : 1/5 [20%]  
[...] local.INFO: ExampleJob [2] RAN  
[...] local.INFO: Batch [9c1b3bef-7582-4007-973c-ba483306290b] progress : 2/5 [40%]  
[...] local.INFO: ExampleJob [3] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1b3bef-7582-4007-973c-ba483306290b] progress : 3/5 [60%]  
[...] local.INFO: ExampleJob [4] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1b3bef-7582-4007-973c-ba483306290b] progress : 4/5 [80%]  
[...] local.INFO: ExampleJob [5] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1b3bef-7582-4007-973c-ba483306290b] progress : 5/5 [100%]  
[...] local.INFO: Batch [9c1b3bef-7582-4007-973c-ba483306290b] ended.

We can see that batch has been cancelled during the second job and each subsequent job din't do anything.
Nice... but they still ran. If, as an example, we have a batch of 5000 jobs, this means that even if cancelled on second job, all 5000 jobs have to run, even if they do nothing. That's lot of useless work.. and time.
We'll see how to overcome this in the next article of the series.

Note that Laravel offer a quick way to not run a job if batch is cancelled by using the SkipIfBatchCancelled middleware:

use Illuminate\Queue\Middleware\SkipIfBatchCancelled;

/**
 * Get the middleware the job should pass through.
 */
public function middleware(): array
{
    return [new SkipIfBatchCancelled];
}

The advantage of using if ($this->batch()->cancelled()) {...} is to have more control of what should be done before bypassing execution. This may be useful when there is a need for a specific log for example.

Managing errors in batches

How error are managed in batches?
Let's find out by making our job fail if it received the number 3.

# app/Jobs/ExampleJob.php

// ...
class ExampleJob implements ShouldQueue
{
  //...

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        t// If batch is cancelled, stop execution
        if ($this->batch()->cancelled()) {
            Log::info(sprintf('%s [%d] CANCELLED: Do nothing', class_basename($this), $this->number));
            return;
        }

        throw_if($this->number == 3, new Exception('I fail because 3.'));
        Log::info(sprintf('%s [%d] RAN', class_basename($this), $this->number));
    }
}

Restart the queue and run it:

sail artisan queue:restart
sail artisan queue:work

Run the batcher:

sail artisan batcher

Batch failed as expected as we can see in logs:

[...] local.INFO: Batch [9c1c043f-18c4-4021-bfd1-c795380748a1] created.  
[...] local.INFO: ExampleJob [1] RAN  
[...] local.INFO: Batch [9c1c043f-18c4-4021-bfd1-c795380748a1] progress : 1/5 [20%]  
[...] local.INFO: ExampleJob [2] RAN  
[...] local.INFO: Batch [9c1c043f-18c4-4021-bfd1-c795380748a1] progress : 2/5 [40%]  
[...] local.ERROR: I fail because 3. {"exception":"[object] (Exception(code: 0): I fail because 3. at /var/www/html/app/Jobs/ExampleJob.php:36)
// ...
[...] local.INFO: ExampleJob [4] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1c043f-18c4-4021-bfd1-c795380748a1] progress : 3/5 [60%]  
[...] local.INFO: ExampleJob [5] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1c043f-18c4-4021-bfd1-c795380748a1] progress : 4/5 [80%]

There's few things to notice here.
First, we don't have the message telling that the batch ended.
Second, progression stops at 4/5 [80%] leaving us to think that the batch didn't ended.
Last but most important, as expected when a job fails, the batch is cancelled and as ExampleJob skips its work when this happened, no other work is done. That may or may not be a desired behavior.

Let's dig deeper on this three topics.

Managing errors in batches: ->then(...) vs ->finally(...)

The message telling that job has ended is missing. Why so?
Answer is simple: ->then(...) callback is called only when all batch's jobs succeeded.
Let's try to remove our Exception and retry the failed job to see what happens.
First, update our ExampleJob:

# app/Jobs/ExampleJob.php

// ...
class ExampleJob implements ShouldQueue
{
  //...

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        t// If batch is cancelled, stop execution
        if ($this->batch()->cancelled()) {
            Log::info(sprintf('%s [%d] CANCELLED: Do nothing', class_basename($this), $this->number));
            return;
        }

        // throw_if($this->number == 3, new Exception('I fail because 3.'));
        Log::info(sprintf('%s [%d] RAN', class_basename($this), $this->number));
    }
}

Restart the queue and run it:

sail artisan queue:restart
sail artisan queue:work

And have the failed job from our batch retried:

sail artisan queue:retry-batch 9c1c043f-18c4-4021-bfd1-c795380748a1

Now, if we check the logs, we see this:

[...] local.INFO: ExampleJob [3] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1c043f-18c4-4021-bfd1-c795380748a1] progress : 5/5 [100%]  
[...] local.INFO: Batch [9c1c043f-18c4-4021-bfd1-c795380748a1] ended.

And this is awkward. Job is correctly retried but as batch is cancelled, work was skipped! At least, progression reach the 100% and we have our final message because all jobs are successful now. The design choice exposed here is that when a batch is cancelled, it should be left as is and not retried in any manner as this won't changed anything, except for the operation that can run when batch ends.
It is possible to alleviate the problem of the missing operation by using the ->finally(...) method, which is run when all jobs of the batch ran once disregarding if they succeeded or failed.
Let's try it. First, restore our ExampleJob:

# app/Jobs/ExampleJob.php

// ...
class ExampleJob implements ShouldQueue
{
  //...

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        t// If batch is cancelled, stop execution
        if ($this->batch()->cancelled()) {
            Log::info(sprintf('%s [%d] CANCELLED: Do nothing', class_basename($this), $this->number));
            return;
        }

        throw_if($this->number == 3, new Exception('I fail because 3.'));
        Log::info(sprintf('%s [%d] RAN', class_basename($this), $this->number));
    }
}

Add the finally() in our batcher:

# app/Console/Commands/Batcher.php

class Batcher extends Command
{
    // ...

    /**
     * Run a batch of ExampleJobs
     *
     * @return void
     */
    private function runExampleBatch(): void
    {
        Bus::batch(
            Arr::map(
                range(1, 5),
                fn ($number) => new ExampleJob($number)
            )
        )
            ->before(fn (Batch $batch) => Log::info(sprintf('Batch [%s] created.', $batch->id)))
            ->then(fn (Batch $batch) => Log::info(sprintf('Batch [%s] ended.', $batch->id)))
            ->progress(fn (Batch $batch) =>
            Log::info(sprintf(
                'Batch [%s] progress : %d/%d [%d%%]',
                $batch->id,
                $batch->processedJobs(),
                $batch->totalJobs,
                $batch->progress()
            )))
            ->finally(fn (Batch $batch) => Log::info(sprintf('Batch [%s] finally ended.', $batch->id)))
            ->dispatch();
    }
}

Restart the queue and run it:

sail artisan queue:restart
sail artisan queue:work

And run a new batch:

sail artisan batcher

And inspect the logs:

[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] created.  
[...] local.INFO: ExampleJob [1] RAN  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] progress : 1/5 [20%]  
[...] local.INFO: ExampleJob [2] RAN  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] progress : 2/5 [40%]  
[...] local.ERROR: I fail because 3. {"exception":"[object] (Exception(code: 0): I fail because 3. at /var/www/html/app/Jobs/ExampleJob.php:36)
// ...
[...] local.INFO: ExampleJob [4] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] progress : 3/5 [60%]  
[...] local.INFO: ExampleJob [5] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] progress : 4/5 [80%]  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] finally ended.

We have the expected message informing us that all the jobs of the ran once.

Managing errors in batches: progression is not at 100%... and is not accurate

Let's inspect the log one more time:

[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] created.  
[...] local.INFO: ExampleJob [1] RAN  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] progress : 1/5 [20%]  
[...] local.INFO: ExampleJob [2] RAN  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] progress : 2/5 [40%]  
[...] local.ERROR: I fail because 3. {"exception":"[object] (Exception(code: 0): I fail because 3. at /var/www/html/app/Jobs/ExampleJob.php:36)
// ...
[...] local.INFO: ExampleJob [4] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] progress : 3/5 [60%]  
[...] local.INFO: ExampleJob [5] CANCELLED: Do nothing  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] progress : 4/5 [80%]  
[...] local.INFO: Batch [9c1c0c83-f801-4025-9e4c-9d85d018dbe8] finally ended.

We already noticed that the progression stops at 80%.
There's 2 issues here:

  • from the batch point of view, we should see 100% as it "ran" completely

  • The 80% takes in account the succeeded jobs... and the job that didn't do anything because batch is cancelled. This means that this 80% tells us: I've passed through 80% of what I've should done but I don't if everything runs effectively!

The first issue can be solved by retrying job which may eventually lead to 100% completion.
Regarding the second issue, we could have jobs throw an Exception if batch is cancelled, leading to all pending job failing if one fails. This is not a good solution as it is exactly what chained jobs are designed for: dependent jobs and if one fails, stop the chain. A better solution would be to avoid batch to be cancelled if a job fails.

Note: it is also possible to simply update the progress callback to have it return a progression that reflects exactly wht we want but we try to stick to Laravel way of doing things in order to fully understand how batch works.

Managing errors in batches: to cancel or not to cancel

When a job fails, the batch is cancelled.
This may not be a desired behavior as this means that some jobs won't do any work. This may be suited when jobs are dependent on each other but if we just want to send emails to different users, we don't want that a failure stops the mail sending.

Laravel offers a quick way to change this behaviour: allowFailures method.
Using this method, if a job fails, it won't be cancelled. In our case, this means our jobs will run completely and batch will reach is end.

Let's update the Batcher command to use it:

# app/Console/Commands/Batcher.php

// ...
class Batcher extends Command
{
  /**
     * Run a batch of ExampleJobs
     *
     * @return void
     */
    private function runExampleBatch(): void
    {
        Bus::batch(
            Arr::map(
                range(1, 5),
                fn ($number) => new ExampleJob($number)
            )
        )
            ->before(fn (Batch $batch) => Log::info(sprintf('Batch [%s] created.', $batch->id)))
            ->then(fn (Batch $batch) => Log::info(sprintf('Batch [%s] ended.', $batch->id)))
            ->progress(fn (Batch $batch) =>
            Log::info(sprintf(
                'Batch [%s] progress : %d/%d [%d%%]',
                $batch->id,
                $batch->processedJobs(),
                $batch->totalJobs,
                $batch->progress()
            )))
            ->finally(fn (Batch $batch) => Log::info(sprintf('Batch [%s] finally ended.', $batch->id)))
            ->allowFailures()
            ->dispatch();
    }
}

And run a new batch:

sail artisan batcher

And check the logs:

[...] local.INFO: Batch [9c1d25a9-c058-4abb-82fd-b8ae09c24084] created.  
[...] local.INFO: ExampleJob [1] RAN  
[...] local.INFO: Batch [9c1d25a9-c058-4abb-82fd-b8ae09c24084] progress : 1/5 [20%]  
[...] local.INFO: ExampleJob [2] RAN  
[...] local.INFO: Batch [9c1d25a9-c058-4abb-82fd-b8ae09c24084] progress : 2/5 [40%]  
[...] local.INFO: Batch [9c1d25a9-c058-4abb-82fd-b8ae09c24084] progress : 2/5 [40%]  
[...] local.ERROR: I fail because 3. {"exception":"[object] (Exception(code: 0): I fail because 3. at /var/www/html/app/Jobs/ExampleJob.php:36)
...
[...] local.INFO: ExampleJob [4] RAN  
[...] local.INFO: Batch [9c1d25a9-c058-4abb-82fd-b8ae09c24084] progress : 3/5 [60%]  
[...] local.INFO: ExampleJob [5] RAN  
[...] local.INFO: Batch [9c1d25a9-c058-4abb-82fd-b8ae09c24084] progress : 4/5 [80%]  
[...] local.INFO: Batch [9c1d25a9-c058-4abb-82fd-b8ae09c24084] finally ended.

We can see that our jobs ran fine and that the batch ended.

Let's inspect our batch in tinker:

> Bus::findBatch('9c1d25a9-c058-4abb-82fd-b8ae09c24084')
= Illuminate\Bus\Batch {#5209
    +id: "9c1d25a9-c058-4abb-82fd-b8ae09c24084",
    +name: "",
    +totalJobs: 5,
    +pendingJobs: 1,
    +failedJobs: 1,
    +failedJobIds: [
      "b7baa334-0b4d-4a6b-877d-d12d334c5bae",
    ],
    +options: [
      "before" => [
        Laravel\SerializableClosure\SerializableClosure {#5171},
      ],
      "then" => [
        Laravel\SerializableClosure\SerializableClosure {#5169},
      ],
      "progress" => [
        Laravel\SerializableClosure\SerializableClosure {#5167},
      ],
      "finally" => [
        Laravel\SerializableClosure\SerializableClosure {#5202},
      ],
      "allowFailures" => true,
    ],
    +createdAt: Carbon\CarbonImmutable @1716489998 {#5164
      date: ...,
    },
    +cancelledAt: null,
    +finishedAt: null,
  }

The cancelledAt is null as expected, but was less expected is the null finishedAt!
For laravel, a batch is finished if all jobs successfully or if it has failed jobs and is cancelled. We can compare this to a cancelled batch:

= Illuminate\Bus\Batch {#5209
    +id: "9c1d283f-66f0-41f6-a91d-17f90d705e51",
    +name: "",
    +totalJobs: 5,
    +pendingJobs: 1,
    +failedJobs: 1,
    +failedJobIds: [
      "6736a446-5e7d-4926-827d-637526d57fc0",
    ],
    +options: [
      "before" => [
        Laravel\SerializableClosure\SerializableClosure {#5171},
      ],
      "then" => [
        Laravel\SerializableClosure\SerializableClosure {#5169},
      ],
      "progress" => [
        Laravel\SerializableClosure\SerializableClosure {#5167},
      ],
      "finally" => [
        Laravel\SerializableClosure\SerializableClosure {#5202},
      ],
    ],
    +createdAt: Carbon\CarbonImmutable @1716490432 {#5164
      date: ...,
    },
    +cancelledAt: Carbon\CarbonImmutable @1716490433 {#5219
      date: ...,
    },
    +finishedAt: Carbon\CarbonImmutable @1716490433 {#5220
      date: ...,
    },
  }

Here, we can see that the finishedAt date is not null.

One way to overcome can be to update the value of job_batches.finished_at via the finally callback. We can use the BatchRepository to do this a clean way:

# app/Console/Commands/Batcher.php

// ...
class Batcher extends Command
{
  /**
     * Run a batch of ExampleJobs
     *
     * @return void
     */
    private function runExampleBatch(): void
    {
        Bus::batch(
            Arr::map(
                range(1, 5),
                fn ($number) => new ExampleJob($number)
            )
        )
            ->before(fn (Batch $batch) => Log::info(sprintf('Batch [%s] created.', $batch->id)))
            ->then(fn (Batch $batch) => Log::info(sprintf('Batch [%s] ended.', $batch->id)))
            ->progress(fn (Batch $batch) =>
            Log::info(sprintf(
                'Batch [%s] progress : %d/%d [%d%%]',
                $batch->id,
                $batch->processedJobs(),
                $batch->totalJobs,
                $batch->progress()
            )))
            ->finally(function (Batch $batch) {
                app()->make(BatchRepository::class)->markAsFinished($batch->id);
                Log::info(sprintf('Batch [%s] finally ended.', $batch->id));
            })
            ->allowFailures()
            ->dispatch();
    }
}

We use app()->make(BatchRepository::class) to get a BatchRepository and use markAsFinished to fill the finished_at date.
We should be very careful if we change the job count, especially the pending_jobs one. Say we manager to get the pending_jobs down to 0 and have a failed job. We can retry the batch with sail artisan queue:retry-batch but then we will witness something weird: progression will be over 100% and pending job count will be -1...
It is important to know where to stop tweaking!

Run a new batch:

sail artisan batcher

Inspect the batch in tinker:

> Bus::findBatch('9c1e037c-4052-4e39-bff1-d2b90ad16609')
= Illuminate\Bus\Batch {#5221
    +id: "9c1e037c-4052-4e39-bff1-d2b90ad16609",
    +name: "",
    +totalJobs: 5,
    +pendingJobs: 1,
    +failedJobs: 1,
    +failedJobIds: [
      "6bf44977-12aa-4dc0-8489-a9c04e13f37a",
    ],
    +options: [
      "before" => [
        Laravel\SerializableClosure\SerializableClosure {#5171},
      ],
      "then" => [
        Laravel\SerializableClosure\SerializableClosure {#5169},
      ],
      "progress" => [
        Laravel\SerializableClosure\SerializableClosure {#5167},
      ],
      "finally" => [
        Laravel\SerializableClosure\SerializableClosure {#5202},
      ],
      "allowFailures" => true,
    ],
    +createdAt: Carbon\CarbonImmutable @1716527214 {#5164
      date: 2024-05-24 05:06:54.0 UTC (+00:00),
    },
    +cancelledAt: null,
    +finishedAt: Carbon\CarbonImmutable @1716527215 {#5219
      date: 2024-05-24 05:06:55.0 UTC (+00:00),
    },
  }

Great, the batch is now in finished state with no pending jobs and we still have access to the detail of the run such a s the failed and pending jobs count. Still we can use sail artisan queue:retry-batch if we want to complete the batch.

There is another way to have our batch in finished state that is more a of a trick but this will allow us to see the last of the available batch callbacks: catch. catch will run when the first failure is detected and only the first one, subsequent ones will not trigger the callback.
The trick is to treat a failed job as if it was successful and to update the failed jobs count.

Let's see the code:

# app/Console/Commands/Batcher.php

// ...
class Batcher extends Command
{
  /**
     * Run a batch of ExampleJobs
     *
     * @return void
     */
    private function runExampleBatch(): void
    {
        Bus::batch(
            Arr::map(
                range(1, 5),
                fn ($number) => new ExampleJob($number)
            )
        )
            ->before(fn (Batch $batch) => Log::info(sprintf('Batch [%s] created.', $batch->id)))
            ->then(fn (Batch $batch) => Log::info(sprintf('Batch [%s] ended.', $batch->id)))
            ->progress(fn (Batch $batch) =>
            Log::info(sprintf(
                'Batch [%s] progress : %d/%d [%d%%]',
                $batch->id,
                $batch->processedJobs(),
                $batch->totalJobs,
                $batch->progress()
            )))
            ->finally(fn (Batch $batch) => Log::info(sprintf('Batch [%s] finally ended.', $batch->id)))
            ->catch(function (Batch $batch, Throwable $e) {
                foreach ($batch->failedJobIds as $failedJobId) {
                    $batch->recordSuccessfulJob($failedJobId);
                    DB::table('job_batches')->where('id', $batch->id)->update(['failed_jobs' => 0]);
                }
                Log::error(sprintf('Batch [%s] failed with error [%s].', $batch->id, $e->getMessage()));
            })
            ->allowFailures()
            ->dispatch();
    }
}

Every failed jobs found will be treated as a successful one via recordSuccessfulJob meaning that all work for updating pending counts, mask as finished will be done if required.
Then we force the failed_jobs to 0 in order to be able to have the catch triggered on next failure as it will be seen as the first one.

Run a new batch:

sail artisan batcher

First, inspect the logs:

[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] created.  
[...] local.INFO: ExampleJob [1] RAN  
[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] progress : 1/5 [20%]  
[...] local.INFO: ExampleJob [2] RAN  
[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] progress : 2/5 [40%]  
[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] progress : 2/5 [40%]  
[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] progress : 3/5 [60%]  
[...] local.ERROR: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] failed with error [I fail because 3.].  
[...] local.ERROR: I fail because 3. {"exception":"[object] (Exception(code: 0): I fail because 3. at /var/www/html/app/Jobs/ExampleJob.php:36)
...
[...] local.INFO: ExampleJob [4] RAN  
[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] progress : 4/5 [80%]  
[...] local.INFO: ExampleJob [5] RAN  
[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] progress : 5/5 [100%]  
[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] ended.  
[...] local.INFO: Batch [9c1e0f38-9e9f-4e20-b1a1-3921e42bd094] finally ended.

There's a lot to notice here:

  • then callback successfully ran because the ended log is present

  • progression is correct

  • we still have the failure logged to be informed of it, it is not hidden in any way

Let's inspect the batch in tinker:

> Bus::findBatch('9c1e0f38-9e9f-4e20-b1a1-3921e42bd094')
= Illuminate\Bus\Batch {#5205
    +id: "9c1e0f38-9e9f-4e20-b1a1-3921e42bd094",
    +name: "",
    +totalJobs: 5,
    +pendingJobs: 0,
    +failedJobs: 0,
    +failedJobIds: [],
    +options: [
      "before" => [
        Laravel\SerializableClosure\SerializableClosure {#5171},
      ],
      "catch" => [
        Laravel\SerializableClosure\SerializableClosure {#5169},
      ],
      "then" => [
        Laravel\SerializableClosure\SerializableClosure {#5167},
      ],
      "progress" => [
        Laravel\SerializableClosure\SerializableClosure {#5202},
      ],
      "finally" => [
        Laravel\SerializableClosure\SerializableClosure {#5221},
      ],
      "allowFailures" => true,
    ],
    +createdAt: Carbon\CarbonImmutable @1716529183 {#5191
      date: 2024-05-24 05:39:43.0 UTC (+00:00),
    },
    +cancelledAt: null,
    +finishedAt: Carbon\CarbonImmutable @1716529184 {#5217
      date: 2024-05-24 05:39:44.0 UTC (+00:00),
    },
  }

As expected, pendingJobs and failedJobs counts are at 0 and batch is in finished state.
What should concern us is the fact that failedJobIds is empty, meaning there is no way to retry the failed jobs. That may be what we want for certain use cases but we have to be aware of that if using this trick. Additionally, using DB::table(...)->update(...) to update database exposes us to breaking change: a change in the database structure will break our code.

Conclusion

This concluded our overview of Laravel batches.
We saw how batches work, what are all the available callbacks.
We also saw that they are flexible enough to allow user to implement their own batch's behavior but we also saw that each change must be done carefully as the openness is not error-proof: you can easily implement a behavior which leads to impossible batch state (pending job count lower than 0) or to state where feature won't work anymore (treat failed jobs as successful ones won't allow to retry them).
This is a general rule of thumb, it is always a good idea to have a clear idea what a feature does and what depends on it before doing any overrides. And beyond the warning, we also need to undestrand our use case in order to know what we need and what we don't. Every choices have advantages and drawbacks and maybe this one feature we lost is useless for us...

In the next article, we will check how multiple batches run in the queue.

0
Subscribe to my newsletter

Read articles from Dominique VASSARD directly inside your inbox. Subscribe to the newsletter, and don't miss out.

job batchingLaravelbatchPHPqueuelaravel 11php8

Written by

Dominique VASSARD
Dominique VASSARD