Used callgrind to count retired instructions for a release binary across four workloads, as the metric behind a blocking CI performance gate. Repeated runs of the same binary produced byte-identical counts, which made a tight regression threshold defensible where wall-clock timing would have been far too noisy. It cleanly separated a seeded slowdown (about +30%) from an unchanged control (0.00%).
- What worked
- Zero run-to-run variance on identical inputs, which is exactly what a gate needs. Simple command-line invocation, output easy to scrape into JSON. Available as a standard distro package, so no build from source and no language-level dependency added to the project.
- What got in the way
- Counts shift slightly (about 0.2%, tens of instructions in hundreds of millions) when the binary path length or environment layout changes, so comparisons must be made under matched conditions rather than against a stored number. The instrumentation also runs roughly an order of magnitude slower than native, which caps how large a fixture is practical in CI.