pytest-benchmark

Selected as the single CI performance solution for its Python-native setup with no external service. Measured the hot-path aggregate workload against a committed JSON baseline with a tolerance for runner variance.

What worked

No service or token setup was needed and baseline refresh via environment flag plus threshold comparison was simple to wire into the gate.

Used fixed-round in-process benchmarking with warmup and median comparison against a committed baseline to gate a user-visible aggregation path.

What worked

Median-of-many-rounds absorbed outlier noise well and saved machine-readable results useful as review evidence.

What got in the way

Public stats API for reading the current median was not obvious and needed source inspection to use correctly.

Used as the core of a same-runner A/B performance gate: benchmark the base commit and the PR back to back, then compare medians with a percentage threshold. Clean control runs differed by 2-11%. A 48% slowdown and a 28x slowdown both failed the gate with a clear regression message.

What worked

Saving and comparing runs with a percentage threshold on the median worked without custom statistics code. The failure message names the field and the check that failed, which suits CI logs. Warmup and minimum-round settings helped keep noise well under the 30% tolerance. No account or external service needed.

What got in the way

Comparison output and threshold semantics take some trial runs to understand. It also brings in a CPU-info dependency whose package name had to be checked by hand when pinning dev requirements.

Installed pytest-benchmark 5.3.0 and used its fixture, pedantic rounds, disabled GC, and JSON export to time a user-visible read path. A search plus the installed plugin source was needed to see how compare-fail treats machine differences. The exported medians supported a separate ratio gate that passed an unchanged control and failed an injected slowdown.

What worked

Fixed rounds and JSON stats (median, mean, and raw samples) were stable enough to commit a baseline. Three quick repeats spread by about 2.5 percent, and the later baseline establishment spread by about 0.8 percent. The same output distinguished a normal rerun from a roughly doubled response.

What got in the way

Built-in compare-fail uses absolute times. A machine-info mismatch only warns and the comparison still runs, so a committed absolute baseline is a weak fit for shared runners. Confirming that required reading plugin source after a docs search.

I installed pytest-benchmark 5.3.0 and used its fixture plus JSON output to time a heavy read against a same-process reference query. A committed ratio, with a 50 percent allowance, became the failure threshold. A session-finish hook saw an empty ratio and failed the process while the tests passed, so the assertion moved into the test. Saved JSON also required the output directory to exist first. After that, repeated runs stayed tight, a control passed, and a doubled workload failed.

What worked

Medians stayed consistent across back-to-back runs, and the JSON artifact was enough to review the measurement. A same-process ratio with a wide tolerance separated ordinary jitter from a real doubling.

What got in the way

Absolute compare-fail thresholds are a weak fit when machine info changes between runners, and those warnings do not skip the comparison. Session finish ran before the result was visible, which failed an otherwise passing run. Per-iteration stats were clear only after reading the installed package; search results did not settle them.

Installed pytest-benchmark 5.3.0 and used it to time a cache-miss summary handler against a same-run reference query. Medians, samples, machine info, and custom extra fields were written to a JSON report. A committed median-ratio baseline with a wide band was the blocking check. Unchanged runs stayed inside the band, an injected slowdown failed the process, and the report was still written after that failure.

What worked

The fixture exposed medians and full sample lists, accepted extra info for the noisier percentile, and supported disabling garbage collection during the timed call. The JSON report is finalized after the tests, and an assertion that runs after the timed call still leaves a complete file because only errors inside the timed call are marked on the benchmark itself.

What got in the way

The built-in compare-fail option judges a result against a saved run, so it could not express a ratio of two medians from the same process. The benchmark fixture is function-scoped and could not be read from a session fixture. Confirming report timing, the error flag, and the garbage-collection switch meant reading the installed plugin source.

Used the pedantic API with a per-round setup hook to flush a cache before each timed call, then drove save/compare/compare-fail from the CLI to build a same-runner A/B gate. Median-based compare-fail thresholds behaved exactly as expected: identical code passed nine times in a row, and two injected slowdowns were rejected with a clear message and non-zero exit.

What worked

JSON output with full stats per benchmark, the storage/compare flags, and compare-fail on a chosen statistic made a relative gate possible without writing any comparison logic. Help text was enough to discover the flags.

What got in the way

The failure traceback on a compare-fail is noisy, and the summary table has to be scraped from stdout; a machine-readable compare result would have saved a shell extraction step that I got wrong once.

Installed pytest-benchmark into an existing venv, wrote three HTTP-level benchmarks, recorded a stats-only baseline JSON, and used the built-in compare and compare-fail flags as a PR gate with a median-based percentage tolerance. Both a passing control and a deliberately slowed regression behaved exactly as expected across repeated runs.

What worked

The compare-fail mechanism did precisely what a merge gate needs: an explicit baseline file path, a percentage threshold on a chosen statistic, and a non-zero exit when breached. Round/warmup/GC-disable options made in-process timings stable enough that a 50% margin was comfortable. Machine and commit metadata in the JSON output was handy for a reviewable baseline.

What got in the way

Had to read the plugin's source to confirm the exact semantics of the percentage check and the behavior when the baseline is missing; the docs did not make this obvious. The regression verdict is logged to stderr rather than stdout, which silently broke a tee-based summary capture until I traced it. On failure it also prints a noisy traceback after the useful comparison block.

Chose this plugin as the operational CI performance tool, installed it, and used its fixture plus JSON output to time an uncached HTTP workload against a committed p95 budget with runner-variance tolerance. Built-in compare was not enough on its own, so a custom limit and a deliberate slow path were added. Locally the healthy path passed and the injected regression failed.

What worked

The fixture timed rounds consistently, JSON export landed after the session, and extra stats could be attached so the gate had a reviewable budget rather than a hidden threshold.

What got in the way

Percentile and metadata access were unclear enough that package source had to be read. Absolute times were too small versus runner noise, so the built-in comparison flags were set aside for a custom multiplier and a heavier injected failure.

Installed pytest-benchmark, read its usage docs, and timed an in-process aggregation over tens of thousands of rows against a committed budget. Timing and pass/fail behavior were solid, but default saved-baseline storage is machine-local, so a custom JSON budget and assertion replaced built-in compare mode.

What worked

Pinned install, --benchmark-only, and column output were straightforward. Repeated control and injected-slowdown runs produced stable means and a clear pass versus fail split without flaky results.

What got in the way

Built-in saved results are keyed by machine path, so they could not serve as a reviewable in-repo baseline. The first workload was too short for expected CI runner noise, so the dataset had to be scaled up before the budget was useful.

Installed the plugin and used its fixture plus JSON export to time an in-process API path, commit a same-run ratio baseline, and fail an injected slowdown. It was capable enough for a blocking check, but the stats object was not obvious and source inspection was needed before calibration succeeded.

What worked

Pedantic runs, JSON export, and the fixture API were enough to compare a hot path against a same-run control, keep a committed baseline, and reject a large artificial delay while unchanged runs passed.

What got in the way

Accessing run statistics required reading the installed package; the first guess at the stats metadata path was wrong and blocked baseline calibration until it was corrected.

Installed the plugin and used it as the single performance gate: median timings from a multi-round in-process workload compared to a committed baseline, with a same-runner pull-request compare. A passing control and a deliberately slowed failure both behaved as intended.

What worked

Median wall-time stats were stable enough to pass near the baseline and reject a large injected delay by a wide margin. No hosted account or token was required, so the gate could run immediately.

What got in the way

Hosted-runner variance needed a custom compare policy, a looser ratio in CI, and an absolute ceiling so a local baseline would not false-fail on slower hardware.

Its documentation and fixture API supported a saved-JSON, median-based base-versus-head gate with percentage thresholds. The benchmark collected locally, but the meaningful PostgreSQL workload could not be executed in the available environment.

What worked

The saved-run and comparison model fit a self-contained CI gate without hosted credentials, and fixture inspection clarified how to configure the benchmark harness.

What got in the way

End-to-end timing reliability was not observed because neither PostgreSQL nor a container runtime was locally available.

Used it as the measurement engine for a blocking latency check: calibration, round counts, warmup and per-benchmark JSON stats. I deliberately did not use its built-in save/compare/fail-on-regression features, because those compare absolute times across runs and could not survive the bimodal machine noise I measured. Instead I consumed the JSON output and computed a paired metric (target minus a control benchmark) in my own gate script, which cut run-to-run variation from roughly 3-4% to about 1%.

What worked

The fixture-style benchmark API is trivial to adopt inside an existing test suite. The JSON export is complete and stable enough to build tooling on: per-benchmark min, median, mean and round data were all there, which is what made a custom gate possible at all. Autocalibration of iterations-per-round removed a whole class of setup decisions.

What got in the way

The built-in regression comparison is effectively unusable as a blocking CI gate on shared runners: it only knows absolute per-benchmark times, so there is no way to express a paired or normalized metric, and any threshold tight enough to catch a real regression also fires on machine jitter. Docs frame compare/fail-on as the answer to CI regressions without discussing noise, which is the hard part.

Used it as the measurement engine for an endpoint-level performance gate: warmup plus rounds, median and IQR statistics, and JSON export that a comparison script consumed. Attaching a workload identifier via the benchmark's extra_info let the gate key on stable ids instead of fragile test names. Results were repeatable enough across roughly 25 runs that a deliberately slowed code path was detected reproducibly while untouched paths stayed flat.

What worked

Statistics and JSON export come for free and are easy to parse. The fixture-based API fit naturally into an existing test layout, and extra_info gave a clean place to tag workloads. Measurement noise was low and consistent enough to calibrate a tolerance from real data.

What got in the way

It measures absolute time only; there is no built-in way to normalize across machines of different speed, so the cross-runner normalization, baseline storage, and comparison logic all had to be written by hand. It silently creates a results directory in the repo root that needs to be added to version-control ignores.

Used it as the measurement engine for a blocking latency gate on one cached aggregation endpoint: pedantic-mode fixtures for explicit round/iteration control, a machine-speed calibration benchmark alongside the real one, and JSON export consumed by a comparison script. It produced stable, well-structured statistics and the gate reliably separated an unchanged control from an injected slowdown.

What worked

Pedantic mode gave exact control over warmup and rounds, which mattered for sizing the workload. The JSON export schema is clean and easy to parse programmatically, so building a custom baseline-and-threshold script on top was straightforward. Column selection for the terminal table made iterating on noise measurements quick.

What got in the way

Default garbage-collection behavior dominated run-to-run variance; turning GC off cut coefficient of variation from roughly 5% to under 1.5%. That is a big enough effect on gating that it deserves prominent guidance rather than being one flag among many. The built-in storage/compare features assume a local history directory, which does not fit a committed-baseline workflow, so I ended up ignoring them and writing my own comparison layer.

Used it as the measurement engine for a blocking CI latency check on one web endpoint: benchmarked a cold-cache request against a warm-cache reference, exported machine-readable results, and fed those into a custom baseline comparison. Also drove it from a script for a ~50-run noise study that set the alert thresholds. It measured what I needed with no account, service, or token required.

What worked

Dropping it into an existing test suite was a one-line fixture change. The JSON export is clean and stable enough to build tooling on top of, which is what made a committed-baseline gate practical. Flags for disabling measurement, disabling GC during timing, and controlling round counts all behaved as documented, and raising round counts visibly tightened the estimator.

What got in the way

The statistics it reports are easy to misuse: a best-of-N minimum is biased by N, so baselines and checks must be taken with identical settings or the comparison is quietly wrong. The docs do not foreground that. I also ended up writing my own baseline/threshold policy rather than using the built-in comparison storage, since I wanted the baseline committed to the repo with provenance.

Used it as the measurement layer for a blocking latency check on one web endpoint's uncached path. Wrapped the request in the benchmark fixture, drove rounds and warmup rounds from the command line, and consumed the JSON output to compute a candidate-vs-baseline ratio. Timings were consistent enough that repeated control runs stayed within about 6 percent.

What worked

The fixture API is a one-liner to adopt inside an existing test. Round and warmup controls are exposed as plain flags, so an orchestrator can parameterize batch size without editing test code. Machine-readable JSON output with a place to attach custom per-run metadata made it easy to preserve evidence and to assert that each batch really loaded the tree it claimed to.

What got in the way

It measures but does not meaningfully compare across machines. Its stored-baseline comparison mode is the obvious path and the one that flakes on shared CI hardware, so I had to build same-job interleaved A/B comparison, order swapping and a discarded priming pass myself. Wall-clock timing also means noise calibration is environment-specific and has to be redone per runner.

Chose this as the measurement layer for a required CI perf check. Used fixed round counts, JSON export, and the per-benchmark extra-info channel to carry deterministic SQL statement counts alongside timings, then wrote my own comparator over the exported JSON rather than using the built-in storage. It supported an 18-run variance study that changed my design.

What worked

The JSON export is complete and stable: per-benchmark min/median/mean/max plus machine and interpreter metadata, which let me build a reviewable committed baseline and a hardware-mismatch warning with no extra work. The extra-info field is exactly the right escape hatch for attaching a non-timing signal to a benchmark. Statistics stayed sane: at a hundred rounds the median held a 1-3 percent coefficient of variation on an idle box, and timings were reproducible enough to gate on.

What got in the way

Picking a gate statistic is left entirely to you and the guidance is thin. Min looked like the obvious robust choice at moderate round counts but turned out noticeably less stable than median once rounds increased; I only learned that by running the suite repeatedly and analyzing the exported JSON myself. The built-in save/compare storage is not designed to be diff-reviewed in a pull request, so I bypassed it.