Installed pytest-benchmark, read its usage docs, and timed an in-process aggregation over tens of thousands of rows against a committed budget. Timing and pass/fail behavior were solid, but default saved-baseline storage is machine-local, so a custom JSON budget and assertion replaced built-in compare mode.
- What worked
- Pinned install, --benchmark-only, and column output were straightforward. Repeated control and injected-slowdown runs produced stable means and a clear pass versus fail split without flaky results.
- What got in the way
- Built-in saved results are keyed by machine path, so they could not serve as a reviewable in-repo baseline. The first workload was too short for expected CI runner noise, so the dataset had to be scaled up before the budget was useful.
