Optimizing Rails Test Speed

TL;DR:

1. If your Rails tests run in the default transaction, disable table truncation using ENV["SKIP_TEST_DATABASE_TRUNCATE"] for faster parallel tests! (Rails 7.2.0+).

2. Make sure your DB files are stored on an SSD to improve integrated test response times.

Fast tests? Yes, please.

I like to have a fast feedback loop on my unit tests. When I'm programming in Ruby, I will either use Guard or Retest to automatically run a test file every time it, or the file it is testing, is saved. This generally provides a very quick feedback loop, even if it is running all of the tests in the file. However, one of my test files began taking 14 seconds to complete, which made it impossible to test small iterations automatically.

I tried disabling blocks of tests and suddenly my test speed increased. I started incrementally adding back tests to see when it would slow back down. It did, but it didn't matter which test I put back, it would instantly slow down as soon as I had more than 50 tests.

I disabled parallel testing to see if it made a difference, and the test ran in just over 1 second. Interesting... but not a good compromise when it will dramatically slow down the entire test suite. So I decided to dig deeper.

Our codebase is using shoulda-matchers to verify that our validations and associations are configured correctly. So for a model with many fields and complicated associations, the number of tests soon exceeded 50, which turns out to be the default trigger for running tests in parallel. So that explained why either reducing the number of tests or disabling parallel testing alleviated the issue. But it left me with the question of why my parallel tests were so slow to start.

The culprit discovered

Within the test.log file I discovered that a TRUNCATE_TABLE command was being run on every database table before the parallel tests began. This step was causing a long delay of around 12 seconds. But was it necessary?

Now that I knew what the source of my pain was, I began doing some research to see if others had run into the same issue. My search took me to an already merged PR #51686 in the Rails repository. It appears DHH himself was being annoyed by a similar issue and had added an ENV flag to disable the TRUNCATE_TABLE command on multi-process test runs. Truncation is only needed if you have tests that don't run inside the default transaction, which is rare. Awesome!

I added the ENV["SKIP_TEST_DATABASE_TRUNCATE"] flag to my local settings and ran the tests again only to watch them continue to run slowly. Hmm... That PR was merged on May 7, 2024, but it turns out that it didn't make it into Rails 7.1.3.4. So I backported the change into my system and tried again. My tests began running quickly, hurrah! (Now that Rails 7.2.0 has been released, new apps will be able to take advantage of this option immediately).

Is it only me?

But now I was curious why no one else on my team had complained about the slow tests. Perhaps I'm the only one that does the instant test run on every file save, but even then a 12 second delay on the test suite starting should have raised some eyebrows. So I asked my colleagues and one of them responded with his own experience, which was that the test in question ran in under 2 seconds, and the test suite also started within just a few seconds. Why the discrepancy?

We were both running PostgreSQL in Docker containers, so I began looking at the underlying hardware. My colleague was using a MacBook with an SSD drive, whereas my Arch Linux desktop has a variety of drives installed. While my code resides on an SSD drive, the docker volumes are stored by default in the /var folder, which in my case is mounted to a rather slow HDD drive.

Lessons learned

Though I wanted some things such as package cache files and logs on the HDD, there are some things in /var that would definitely benefit from faster access. So this has given me a reason to take a look at my system configuration and look into making some adjustments to improve performance. Nevertheless I learned some interesting things by going down this rabbit hole, and perhaps my experience will help others hoping to optimize their development experience.