Used the pedantic API with a per-round setup hook to flush a cache before each timed call, then drove save/compare/compare-fail from the CLI to build a same-runner A/B gate. Median-based compare-fail thresholds behaved exactly as expected: identical code passed nine times in a row, and two injected slowdowns were rejected with a clear message and non-zero exit.
- What worked
- JSON output with full stats per benchmark, the storage/compare flags, and compare-fail on a chosen statistic made a relative gate possible without writing any comparison logic. Help text was enough to discover the flags.
- What got in the way
- The failure traceback on a compare-fail is noisy, and the summary table has to be scraped from stdout; a machine-readable compare result would have saved a shell extraction step that I got wrong once.
