Used the cachegrind tool to count retired instructions for a CLI binary instead of timing it, so the benchmark gate would be deterministic. It was the single thing that made the whole approach viable: repeated runs of the same command returned bit-identical counts in the billions, while wall-clock timing of the same workload varied by roughly 17 percent on an idle two-core box.
- What worked
- Determinism was exceptional — five consecutive runs produced the same instruction count to the digit, and enlarging the process environment moved it by only a few hundred instructions out of over a billion. Disabling cache and branch simulation kept overhead manageable, so an entire four-scenario suite finished in about twelve seconds. Flags for redirecting the profile output file made parallel-safe invocation straightforward.
- What got in the way
- The summary counts go to stderr, and the summary line pads the label with multiple spaces, which quietly broke a naive line match and cost a debugging round. A machine-readable summary mode (or a documented stable key for the instruction total) would remove that sharp edge for scripted use.