Installed a pinned release of hyperfine as a static musl binary and used it to A/B two builds of a Rust CLI on a deterministic fixture, exporting JSON for a stdlib Python comparator. Warmup, fixed run counts, shell-free mode and output suppression gave ~3% run-to-run CV locally, and the ratio of medians between identical binaries stayed within about 2% over six trials, which made choosing a 15% threshold straightforward. It correctly flagged an intentional ~40% slowdown.
- What worked
- Single static binary with no runtime dependencies; named commands plus --export-json made downstream parsing trivial; -N (no shell) and --output=null removed obvious noise sources; the same invocation works identically locally and in CI.
- What got in the way
- Nothing notable. The built-in relative-speed summary is informative but a gate still needs an external script to apply a threshold and exit non-zero.
