Installed benchstat to compare two sets of Go benchmark samples and emit statistically qualified deltas. Its text report and -format csv output provided the center, confidence interval, percentage delta and p-value per metric, which I parsed with awk to drive a pass/fail verdict. It correctly flagged a 5% drift between sequential runs as significant and reported identical-code control runs as within noise once builds were made reproducible.
- What worked
- The statistical output is exactly what a regression gate needs: a p-value plus a delta, so thresholds can require both magnitude and significance. CSV mode was clean and machine-friendly. Running it via go run with a pinned version avoided a separate install step in CI.
- What got in the way
- The latest module revision required a newer Go release than the project uses, so go install failed until I pinned an older commit found by listing module versions and inspecting go.mod files; there was no obvious compatibility note. The CSV layout (section headers pairing file names, then metric rows) is not documented in the help text, so I had to run it on real data to learn the column structure before parsing.
