Measure What Matters 🧐

There is a wide variety of improvements engineers can contribute to. These could be improvements in the workflow, cost-savings and often new features to the product. Some tangible numbers connect the business with the value of the improvements engineers are working on. This is great. All engineers should strive to be as metric-driven as possible in everything they do.

But things can still get slippery. There is a joke in the industry that statistics is “the art of deceiving yourself with highly precise numbers”. The more time I spend in the industry, the less this feels like a joke. It’s still funny, though.

In this post we’ll explore how entirely valid metrics can sidetrack us on our way to delivering the highest value. Behind all numbers, something much more important must be present - meaningful impact. As always, we’ll explore what separates the best from the rest.

The metric trap

The team was highly motivated and had their eyes on a target - reduce the end-to-end latency of their platform by 10ms. This is a 50% improvement over the existing 20ms!

They did their due diligence — fully grasped the system, identified an opportunity, drafted a proposal and got it approved. And from that point on, they were on their way. A few months of development, and three senior engineers later - the latency was cut in half. The team is very proud, as they made the system measurably better. Or so they thought.

The latency was significantly improved - so what? Not trying to be rude, but practical. What does this improvement really mean for the system, the business and profits? Without a doubt, a measurable improvement was made. However, they are still left to prove that this particular system improvement has enabled a business improvement.

What if no one cared about the latency being 20ms? What if it was good enough? What if this segment of the platform can function with a 100ms latency and not a single customer would notice the difference? What if the team spent the last few months chasing a number no one cares about being improved?

Metrics are not equivalent to results. They represent a measurable aspect of results, but the relevant, bottom-line results are much more nuanced.

The alternative

Let’s explore how the best engineers avoid this trap. As you can tell by the example above, we were able to discard the improvement relevance by asking many somewhat vague and diverse questions. Yet the questions had a theme - searching for practical value to the business.

I’ve spent a lot of time trying to gauge how top engineers make this distinction. What is the razor they use to cut through the fluff and see the true relevance? I have finally found it - in medicine.

There is a difference between statistically significant results and clinically significant results. The idea is the following: improvements are only valuable if they make a difference to the bottom line.

For example, you buy a new phone charger that is able to charge your phone 200% faster. But the phone itself can only accept power at a fixed rate, rendering the improvement useless. Within a context.

Another example would be improving a manufacturing process in a factory, while the shipping and delivery are the bottlenecks causing the delay. Improvements are very real, but the bottom line did not move an inch. In a different context, this may help the business. In this one it doesn’t.

Top engineers refuse to deceive themselves. They refuse to improve for the sake of improving. They understand that effort spent in one avenue is effort taken away from all others - so it really matters how they allocate it. Measure, don’t guess. But always keep your eyes on the bottom line.

The takeaway

• The best engineers optimise for the business, not just metrics.

• Success isn’t about moving numbers—it’s about making a difference.

• Think about statistical significance and clinical significance when making decisions what to focus on.

Frequently ask “why”, “so what”, “how does it relate”, “does it matter”, etc.