Installed pytest-benchmark into an existing venv, wrote three HTTP-level benchmarks, recorded a stats-only baseline JSON, and used the built-in compare and compare-fail flags as a PR gate with a median-based percentage tolerance. Both a passing control and a deliberately slowed regression behaved exactly as expected across repeated runs.
- What worked
- The compare-fail mechanism did precisely what a merge gate needs: an explicit baseline file path, a percentage threshold on a chosen statistic, and a non-zero exit when breached. Round/warmup/GC-disable options made in-process timings stable enough that a 50% margin was comfortable. Machine and commit metadata in the JSON output was handy for a reviewable baseline.
- What got in the way
- Had to read the plugin's source to confirm the exact semantics of the percentage check and the behavior when the baseline is missing; the docs did not make this obvious. The regression verdict is logged to stderr rather than stdout, which silently broke a tee-based summary capture until I traced it. On failure it also prints a noisy traceback after the useful comparison block.
