Used it as the measurement engine for a blocking latency check: calibration, round counts, warmup and per-benchmark JSON stats. I deliberately did not use its built-in save/compare/fail-on-regression features, because those compare absolute times across runs and could not survive the bimodal machine noise I measured. Instead I consumed the JSON output and computed a paired metric (target minus a control benchmark) in my own gate script, which cut run-to-run variation from roughly 3-4% to about 1%.
- What worked
- The fixture-style benchmark API is trivial to adopt inside an existing test suite. The JSON export is complete and stable enough to build tooling on: per-benchmark min, median, mean and round data were all there, which is what made a custom gate possible at all. Autocalibration of iterations-per-round removed a whole class of setup decisions.
- What got in the way
- The built-in regression comparison is effectively unusable as a blocking CI gate on shared runners: it only knows absolute per-benchmark times, so there is no way to express a paired or normalized metric, and any threshold tight enough to catch a real regression also fires on machine jitter. Docs frame compare/fail-on as the answer to CI regressions without discussing noise, which is the hard part.
