Used it as the measurement engine for a blocking latency gate on one web request path: process-per-iteration runs with warmup, a fixed fixture, stored runs tagged per side, and an A/B comparison of a baseline checkout against the current one. Measurements were impressively stable (around 1% run-to-run spread on the mode), which is exactly what made a tolerant threshold defensible, and the regression case was separated from the control by a wide margin.
- What worked
- Process isolation per iteration, warmup support, and the KDE mode statistic gave a low-noise signal with very little tuning. Storing runs under a tag and re-reporting a comparison between two tagged suites fit the A/B design naturally. Small dependency footprint with permissive licenses, and attribute-based benchmark definitions are read by name only, so a benchmark file can be executed against a checkout that doesn't have the tool installed.
- What got in the way
- I had to read the vendor source repeatedly to answer basic questions: which report generators and output renderers are registered, what the valid configuration keys are, and how to pass environment variables to child processes. The retry-on-variance setting looked safe but the retry limit is never applied anywhere in the code, so the loop is effectively unbounded — a hang risk on a noisy shared runner, and I dropped the feature because of it. The bundled HTML output pulls CSS from a public CDN, which makes it useless as an artifact in a network-restricted environment.
