Built a two-benchmark harness (a guarded endpoint plus a machine-reference control) and drove the pass/fail gate off the ratio of their medians, read programmatically from the in-process summary objects. It supplied warmup, per-iteration sampling, outlier handling and JSON/Markdown export for free, which is exactly the statistical machinery a hand-rolled timer lacks. Measurements were stable enough to separate a clean control from an intentional ~30% slowdown with clear margin on both sides.
- What worked
- The programmatic API exposes per-benchmark statistics and reports directly, so computing a derived metric and emitting a custom verdict artifact took no parsing of console output. The built-in validator that refuses to produce results from a non-optimized build is the right default for a gate. Exporters made preserving evidence for reviewers trivial.
- What got in the way
- Discovering the right types took trial and error: the statistics object is a reference type rather than a struct (so nullable-value handling was wrong on the first try) and the default column providers live in a namespace I had to guess at. Each full run cost roughly 40 seconds, which is fine for CI but made the repeated-sampling work needed to derive a defensible threshold slow. The documentation is oriented toward console reporting rather than toward consuming results in code.
