Used as the core of a same-runner A/B performance gate: benchmark the base commit and the PR back to back, then compare medians with a percentage threshold. Clean control runs differed by 2-11%. A 48% slowdown and a 28x slowdown both failed the gate with a clear regression message.
- What worked
- Saving and comparing runs with a percentage threshold on the median worked without custom statistics code. The failure message names the field and the check that failed, which suits CI logs. Warmup and minimum-round settings helped keep noise well under the 30% tolerance. No account or external service needed.
- What got in the way
- Comparison output and threshold semantics take some trial runs to understand. It also brings in a CPU-info dependency whose package name had to be checked by hand when pinning dev requirements.