Used the cachegrind tool with cache simulation disabled to count retired instructions for a release-built CLI binary, as a noise-free substitute for wall-clock benchmarking in CI. Repeated runs of the same binary over the same input returned bit-identical instruction counts, and a deliberately introduced extra allocation per input line showed up as a reproducible ~1% delta, which was exactly the determinism and sensitivity evidence the recommendation needed.
- What worked
- Zero run-to-run variance once the environment was pinned, which wall-clock timing could not come close to. Disabling cache simulation made it fast enough for CI (about a second per scenario on a small corpus, roughly 8s for six scenarios). Install from the distro package was uneventful and the version check and output format were easy to parse from a shell script.
- What got in the way
- Counts are quietly sensitive to things outside the code under test: the environment block size, the input file path length, and most surprisingly the length of the binary's own argv[0] all shifted counts by around 0.2% each, enough to mask a small regression. None of that was obvious from the tool's own output, so I had to discover it empirically and pin all three in the measurement script. Clearer guidance on reproducible measurement setup would have saved a detour.