The first rewrite failed for a reason that took us eleven months to name. We were optimising throughput — jobs completed per minute — because throughput was the number on the dashboard, and the dashboard had been built by someone who left before any of us joined.

Throughput was never the constraint. The constraint was queue depth at the ninety-ninth percentile, which is a different shape of problem entirely, and one that gets worse precisely when throughput looks healthiest.

What follows is the instrumentation we should have written on day one, and the three graphs that made the answer obvious once we finally drew them.