Used it as the measurement engine for a latency gate on one authenticated page: a bench class with warmup/revs/iterations attributes booting a real application kernel, plus an XML result dump that a wrapper script parsed into a normalised ratio. Measurement quality was good — within-run relative standard deviation stayed at 1-2% once the harness was correct, and it caught an intentionally injected slowdown at roughly +21% while passing unchanged code.
- What worked
- Attribute-based configuration of warmup, revisions, iterations and retry threshold is concise and expressive. Machine-readable XML dump with per-iteration timings made it easy to compute my own medians instead of fighting the built-in reports. Startup cost was low — a full suite ran in a couple of seconds, cheap enough to repeat several times per gate invocation.
- What got in the way
- Config keys were hard to discover from docs; I ended up reading the bundled JSON schema shipped in the package to find valid options. One run produced no output at all and a non-zero exit until I re-ran at high verbosity to see the real error — a silent failure mode. The runner executes all iterations of one subject and then all of the next, so a normalisation ratio between two subjects compares non-adjacent time windows; on a contended machine that produced a false regression and I had to build pairing into a wrapper script.
