Used the cachegrind tool to count executed instructions for a compiled CLI binary across six workload scenarios, as the measurement basis for a blocking CI performance gate. Repeated runs of the same binary produced bit-identical instruction counts, which was the whole reason a hard-fail threshold was defensible; wall-clock timing on the same machine varied by nearly 30 percent run to run.
- What worked
- Perfect run-to-run determinism on an unchanged binary, including across a full recompile from untouched sources. Disabling cache and branch simulation kept overhead low enough to measure a 20k-line corpus in about a second per scenario. Instruction-count deltas tracked measured wall-clock deltas closely on a deliberately slowed build, so it was a faithful proxy. Binary path length shifted counts by only about a hundredth of a percent.
- What got in the way
- The summary counts come out on stderr in a format that has to be scraped with a regex rather than being available as machine-readable output, so the harness depends on a text layout that could change between versions. The simulation slowdown also means corpus size has to be chosen deliberately rather than just reusing a realistic-size input.