Open source: Testing the testers

Node.js latest releases have been quite fun as of late, and jest my go to testing framework wasn’t keeping up with the ecosystem so I thought it was time for me to go back to the good old days of hot drinks testing frameworks, but what was supposed to be a nostalgia trip back to mocha.js actually helped me feel open source in the flesh, open source tools you use everyday are not magical bug-free black boxes. Conceptually sounds obvious but it feels different when it happens to you and find a bug in a tool you used so long.

The first step of TDD is to write the failing test, but what if the test not even gets detected? That was the start of my journey. After spending a good afternoon after trying to track a bug down to a silly lack of environment variable I noticed that was something very fishy about mocha, it never reported top level import errors, an exception was being thrown during module import but mocha was not reporting on it, just saying “0 tests”, I initially didn’t want to blame it on mocha, I like living on the edge and was running the latest node (24) with native type-stripped ESM typescript files. But although I acknowledged the fact that the problem was somewhat self-inflicted I still believed it was relevant enough to raise an issue in mocha’s repository so they could prepare for when those features reach the masses.

I don’t know about you but personally my biggest struggle with open source has always been the social management aspect of working in public: piles of irrelevant issues, sometimes entitled people, finally you find something to contribute on, open a PR about it and have petty developers nitpick on your variable names. So for me I got quite the rush to have my reported bug not only confirmed but also affecting LTS, having that easy of a score I couldn’t resist, I dug in.

Then I realized that a decade old project might not be as accessible to contribute as one initially though, how do you even write tests on a testing framework? CONTRIBUTING.md files are something you can’t quite value until you need one, you can know conceptually that is supposed to help you understand how to contribute, but I hadn’t realized contributing might also mean working on a bug affecting you too, not a document for select few.

Turns out my bug could be tested using mocha’s “integration” test which is somewhat simpler then the unit tests (turns out is kinda complicated to test a test framework) which mostly means spawning a child process of mocha running a “fixture” test and asserting on std-out, exit code, etc.

I’m not particularly religious about TDD but it goes a long way conserving your sanity when trying to find your way in such a big project, secondly knowing my way around a debugger really turned out invaluable, much better to simply drop breakpoints, know all variable values at any given point and step over from there. Codebase which may I remind you, is mostly ES5 JavaScript so that means: no typescript, prototype based OOP, and node callback hell, the code is older than promises so lets just say the age is starting to show.

And I don't mean to complain, mocha is still wildly used and the contributors are right now working to reboot development and if anything all this tech debt just makes me feel I can make a difference, how many nerds are still around to refactor function.proptotype.instanceMethod = function () {} away? Endless contribution opportunities, and mocha still has 22k stars.

After stepping in, out and over, I finally came across a piece of code that although quite clever, assumes a little much about node, and exposed mocha to all kinds of funny bugs from downstream dependencies, but I’m getting ahead of myself, the code goes a little something like this: (overly simplified of course)

async function loadTestFile(resolvedFilePath) {
    // [...]
    try {
        return require(resolvedFilePath);
    } catch (err) {
        return await import(resolvedFilePath);
    }
}

Its quite the ingenious way to support ESM and CJS modules transparently: tries the path of least resistant first by requiring it, if it fails, try falling back to importing, with the big assumption being that if the error thrown by the require wasn’t related to ESM / CJS integration, the second import will simply throw the same error again. Well turns out I had just struck gold because in fact the import call was not throwing the second time! The same module, with the same top level exception, simply disappeared swallowed by node, which was returning a meaningless empty object instead.

Stay tuned to learn how I reported and almost landed a the fix on node.JS main.