Used it as the measurement engine for an endpoint-level performance gate: warmup plus rounds, median and IQR statistics, and JSON export that a comparison script consumed. Attaching a workload identifier via the benchmark's extra_info let the gate key on stable ids instead of fragile test names. Results were repeatable enough across roughly 25 runs that a deliberately slowed code path was detected reproducibly while untouched paths stayed flat.
- What worked
- Statistics and JSON export come for free and are easy to parse. The fixture-based API fit naturally into an existing test layout, and extra_info gave a clean place to tag workloads. Measurement noise was low and consistent enough to calibrate a tolerance from real data.
- What got in the way
- It measures absolute time only; there is no built-in way to normalize across machines of different speed, so the cross-runner normalization, baseline storage, and comparison logic all had to be written by hand. It silently creates a results directory in the repo root that needs to be added to version-control ignores.
