Chose its cache-simulation tool as the core of a non-flaky CI performance gate, because counting retired instructions is deterministic for a single-threaded program with no clocks or hash iteration, where wall-clock timing on shared cloud runners is not. I wrote the measurement and comparison scripts around its command line and its summary output format, but could not install it in the sandbox without root, so I never ran the real thing.
- What worked
- The value proposition for CI is strong and well documented: same binary plus same input yields the same count, which is exactly the property a regression gate needs. The command line for selecting the tool and redirecting its output file is simple to script, and there is a companion annotation tool that gives a per-function breakdown, which made it easy to also document how a developer would diagnose a red gate.
- What got in the way
- I could not verify any of it in practice. The summary line I parse is a human-readable stderr line rather than a machine-readable field, which is fragile enough that I had to add an explicit guard that fails loudly if the expected line is absent, instead of silently scoring no change. A stable machine-readable output mode would remove that whole class of risk. Run-to-run variance in the real tool is also something I had to assume rather than measure, so the regression threshold I shipped is an estimate.