Benchmarking code with PHPBench

As I continue to work on the project to evaluate the recommended maximum size of data that we can manage in a coreBOS install. I started the next steps of constructing a profiling and performance infrastructure for the project so we can analyze objectively how the application performs with a database of 32 million records and 5000 users.

Studying the different options, I found PHPBench, a benchmark runner for PHP analogous to PHPUnit but for performance, and decided to give it a try.

The documentation is spot-on, clear, direct, and just what you need. Getting the tool configured and working is a breeze.

The set of options and functionality is amazing and simple. It just works.

After reading the documentation I decided to follow the same approach coreBOS already has with the unit test project, because it is a separate project that you can apply on any coreBOS install that needs to do the unit tests and I understand that we will be doing performance benchmarking also only on a handful of installs.

So I created a separate repository and downloaded the PHPBench phar file into it.

curl -Lo phpbench.phar https://github.com/phpbench/phpbench/releases/latest/download/phpbench.phar

I validated it as per the instructions and pushed it to the repository.

Now we can start benchmarking some code.

The first challenge is to load the coreBOS infrastructure so that we can access all the awesome functionality. Turns out that the test project also had that problem and solved it by creating a generic "load everything" script. So I just copied that one from there and included it at the top of the benchmark script.

I followed the Quick Start instructions and create this script that executes two functions, 5 times in batches of 1000 runs.

<?php
include_once 'build/evBench/loadcorebos.php';

class CommonUtilsBench {

    /**
    * @Revs(1000)
    * @Iterations(5)
    */
    public function benchgetCurrencyName() {
        getCurrencyName(1, true);
    }

    /**
    * @Revs(1000)
    * @Iterations(5)
    */
    public function benchgpopup_from_html() {
        popup_from_html('$string', true);
    }
}

I ran the benchmarks with this command

build/evBench/phpbench.phar run build/evBench/include/utils/CommonUtils.php --report=aggregate

and received this output

PHPBench (1.2.10) running benchmarks... #standwithukraine
with PHP version 8.2.7, xdebug ✔, opcache ❌

\CommonUtilsBench

    benchgetCurrencyName....................I4 - Mo1.801μs (±9.46%)
    benchgpopup_from_html...................I4 - Mo1.729μs (±1.13%)

Subjects: 2, Assertions: 0, Failures: 0, Errors: 0
+------------------+-----------------------+-----+------+-----+----------+---------+--------+
| benchmark        | subject               | set | revs | its | mem_peak | mode    | rstdev |
+------------------+-----------------------+-----+------+-----+----------+---------+--------+
| CommonUtilsBench | benchgetCurrencyName  |     | 1000 | 5   | 21.985mb | 1.801μs | ±9.46% |
| CommonUtilsBench | benchgpopup_from_html |     | 1000 | 5   | 21.985mb | 1.729μs | ±1.13% |
+------------------+-----------------------+-----+------+-----+----------+---------+--------+

That simple! Really, really nice!

Assertions

Another awesome feature that I found was Assertions. You can annotate your PHPBench scripts with a powerful expression language to validate the timing of your functions and then add these benchmarks to your CI/CD process to detect variations in the execution time of the critical functions of your code base.

I tried that by adding this validation to the two functions in the script above.

* @Assert("mode(variant.time.avg) < 200 ms")

and received this output

    benchgetCurrencyName....................I4 ✔ Mo1.826μs (±2.06%)
    benchgpopup_from_html...................I4 ✔ Mo1.757μs (±8.47%)

Subjects: 2, Assertions: 2, Failures: 0, Errors: 0
+------------------+-----------------------+-----+------+-----+----------+---------+--------+
| benchmark        | subject               | set | revs | its | mem_peak | mode    | rstdev |
+------------------+-----------------------+-----+------+-----+----------+---------+--------+
| CommonUtilsBench | benchgetCurrencyName  |     | 1000 | 5   | 21.985mb | 1.826μs | ±2.06% |
| CommonUtilsBench | benchgpopup_from_html |     | 1000 | 5   | 21.985mb | 1.757μs | ±8.47% |
+------------------+-----------------------+-----+------+-----+----------+---------+--------+

Notice the new (green) line above the table

Subjects: 2, Assertions: 2, Failures: 0, Errors: 0

Next, I tried to force an error with this assertion

* @Assert("mode(variant.time.avg) < 1 microsecond")

and got this response

    benchgetCurrencyName....................I4 ✘ Mo1.791μs (±2.95%)
    benchgpopup_from_html...................I4 ✘ Mo1.726μs (±24.63%)

2 variants failed:

  ✘ \CommonUtilsBench::benchgetCurrencyName # 

    1) mode(variant[time][avg]) < 1 microsecond
       = 1.791373776908 < 1 microsecond
       = false

  ✘ \CommonUtilsBench::benchgpopup_from_html # 

    1) mode(variant[time][avg]) < 1 microsecond
       = 1.726025440313 < 1 microsecond
       = false

Subjects: 2, Assertions: 2, Failures: 2, Errors: 0
+------------------+-----------------------+-----+------+-----+----------+---------+---------+
| benchmark        | subject               | set | revs | its | mem_peak | mode    | rstdev  |
+------------------+-----------------------+-----+------+-----+----------+---------+---------+
| CommonUtilsBench | benchgetCurrencyName  |     | 1000 | 5   | 21.986mb | 1.791μs | ±2.95%  |
| CommonUtilsBench | benchgpopup_from_html |     | 1000 | 5   | 21.985mb | 1.726μs | ±24.63% |
+------------------+-----------------------+-----+------+-----+----------+---------+---------+

Simple and effective!

This is what the final code looked like

    /**
    * @Revs(1000)
    * @Iterations(5)
    * @Assert("mode(variant.time.avg) < 20 ms")
    */
    public function benchgetCurrencyName() {
        getCurrencyName(1, true);
    }

Hard Part

Now we have the infrastructure to measure the performance of functions and processes inside the coreBOS application, but we have to continue asking ourselves, "Which are the methods and processes that we should measure?", "Where are the bottlenecks of our code?", "What code do we have to keep under control?"

Stay tuned as I continue to create the infrastructure to answer those questions.

Thanks for reading.

References

• coreBOS

• PHPBench

• PHPBench Documentation

• coreBOS Benchmark repository