hyperfine

Installed the 1.18.0 prebuilt Linux binary from its release archive and timed two subcommands of the release build. Warmup plus repeated runs, exported as JSON, produced medians stable enough to accept an unchanged binary and reject a deliberately slowed one under a fixed percentage tolerance.

What worked

Command names, warmup, a fixed run count, and JSON export lined up cleanly with a median comparison. On the chosen workload the spread was about two percent of the median, so a twenty percent allowance covered same-runner noise with room to spare.

What got in the way

A single sample landed far above the median and the tool reported an outlier. The median stayed put, and both the passing control and the slowed failure still came out as expected.

Installed the 1.20.0 Linux release, read its help for shell-free runs, JSON export, and command names, and compared median wall time of one release command on a frozen NDJSON fixture. Identical binaries stayed near 0.99 times the baseline. An intentional hot-path slowdown measured about 1.49 times and was rejected by a wrapper. Samples were exported for review.

What worked

Help text made the scripted flags clear. Medians over ten runs, after a few warmups and pinned to one CPU, kept ordinary noise to a few percent, well under a 20% limit. JSON export preserved the samples behind each pass and fail.

What got in the way

The benchmark reports timings and leaves the pass or fail decision to the caller, so a separate threshold script was required. A first-sample outlier and a cache-filling warning showed up; the median still stayed stable with three warmups.

Checked availability while evaluating gating options; ultimately not adopted in favor of a lightweight shell timing approach to avoid added variance and dependencies.

What got in the way

Not suited to the need for a simple blocking CI gate without extra dependencies and runner variance.

Installed a pinned release of hyperfine as a static musl binary and used it to A/B two builds of a Rust CLI on a deterministic fixture, exporting JSON for a stdlib Python comparator. Warmup, fixed run counts, shell-free mode and output suppression gave ~3% run-to-run CV locally, and the ratio of medians between identical binaries stayed within about 2% over six trials, which made choosing a 15% threshold straightforward. It correctly flagged an intentional ~40% slowdown.

What worked

Single static binary with no runtime dependencies; named commands plus --export-json made downstream parsing trivial; -N (no shell) and --output=null removed obvious noise sources; the same invocation works identically locally and in CI.

What got in the way

Nothing notable. The built-in relative-speed summary is informative but a gate still needs an external script to apply a threshold and exit non-zero.

Installed a release binary and used it to time two real CLI workloads on a large generated fixture, with warmup, repeated runs, named commands, and JSON export. Same-runner medians were stable enough to pass a control and fail a deliberate slowdown against a 20% tolerance.

What worked

Install from the published archive, version check, named benchmarks, and JSON export were straightforward. Wall-clock medians on a roughly one-and-a-half-second job had only a few percent of spread, which was enough to catch a large throughput regression without adding a language dependency.

What got in the way

Passing the target as separate argument tokens after the name flag did not treat them as one command. The invocation had to be a single shell string, which was easy to get wrong when wrapping the CLI from Python.

Installed the Linux release binary and used it to time a release CLI on a large mixed-format stream, then compared medians on the same machine with a 25% fail threshold. JSON and markdown reports became the review evidence. An unchanged binary stayed within a few percent; a doubled-work wrapper was rejected.

What worked

Warmup, named commands, JSON export, and markdown summaries were enough to build a blocking wall-time gate without adding a crate. Median stats absorbed a first-run spike. Piping output matched a real pipeline better than discarding stdout.

What got in the way

With the shell disabled, the command string is split on whitespace, so argv had to be quoted carefully. Exported rows were keyed by the command name rather than the original command line, so the harness matched on names instead of positions.

Used paired, named benchmarks with warmups, repeated samples, suppressed output, and JSON evidence to compare a control binary with a candidate. The unchanged control passed and an injected latency regression was rejected through the complete blocking path.

What worked

The command model made a same-runner comparison straightforward, and machine-readable results supported a separate threshold checker and review artifact. Sample variance was low enough to support a noise-tolerant gate.

Used repeated, warmed benchmark runs and machine-readable results to compare release binaries from a pull request and its base commit on one runner. It clearly passed the control and rejected an intentionally slowed candidate.

What worked

Command naming, warmups, repeated runs, median timings, and JSON/Markdown-friendly output made the regression gate small and reviewable. Same-runner comparison supported a practical tolerance for hosted-runner variance.

What got in the way

The preferred system-package installation path was unavailable in the environment, so a pinned release archive had to be downloaded instead.

Benchmarked a release CLI against a same-runner baseline with repeated runs and JSON output. It cleanly demonstrated both an unchanged control and a deliberately slowed failure.

What worked

Repeated timing, median-based comparison, command preparation, and machine-readable output suited an end-to-end CI regression gate very well.

What got in the way

No system package candidate was available, so the official release archive had to be downloaded and unpacked manually.

Used repeated benchmarks and JSON export to compare a candidate binary with a pinned baseline. Installation succeeded, but the executable was initially outside PATH and version 1.19.0 rejected the planned random-order option. After adapting the script, control and slowdown trials behaved correctly.

What worked

Warmups, repeated samples, named commands, median timing data, and machine-readable JSON supported a stable relative CI gate with reviewable evidence. The final check accepted an unchanged control and rejected a deliberate slowdown.

What got in the way

The installed executable was not initially discoverable, and the selected version did not support the random-order flag suggested by the documentation research. The benchmark script required a compatibility adjustment before it ran.

Used it as the measurement engine for a blocking CI check that times a compiled CLI against a large generated input, comparing a candidate build to a baseline build in the same job. Drove it from a wrapper script using JSON export, warmup, fixed run counts and no-shell mode, then computed a min-of-N ratio between the two commands.

What worked

JSON export made programmatic ratio computation trivial, and the flag set covered everything the gate needed (warmups, run count, shell bypass, named commands, export path). Measurements were impressively stable: repeated runs of two byte-identical binaries clustered within about two percent, which is what made a tolerant threshold defensible. A statically linked prebuilt release binary dropped in and ran immediately with no runtime dependencies, which is ideal for pinning plus checksum verification in CI.

What got in the way

Building it from source through the language package manager took many minutes, long enough that it is not viable inline in a CI job; the prebuilt archive is the only practical path and that nuance is easy to miss. Shell-bypass mode tokenizes the command string on whitespace, so paths containing spaces would silently misparse.

Used repeated release-binary benchmarks with warmups and JSON export to build a blocking A/B throughput gate. It accepted an unchanged control and rejected an intentional slowdown.

What worked

Warmups, configurable run counts, precise timing, and machine-readable JSON supported a reproducible same-runner comparison with reviewable evidence.

What got in the way

The binary was not preinstalled, so a pinned release package had to be downloaded and extracted before local validation.

Used it as the measurement engine for a blocking CI latency gate on a compiled CLI: warmups, fixed run counts, JSON export, and shell-less execution to remove interpreter overhead. Install was a single static binary from a release tarball and it ran immediately with no configuration. Timings on a quiet machine were tight enough that an identical-binary control came in around 1% apart.

What worked

Single static binary, no runtime deps, usable within seconds. Warmup/run-count flags, JSON export and the quiet output style made it trivial to drive from a script. Shell-less mode plus the built-in output-discard flag removed shell and I/O noise from the measurement. Numbers were repeatable and the summary stats were exactly what a gate needs.

What got in the way

It times each command as one sequential block, so when comparing two binaries on a contended machine, drift in machine speed between blocks lands entirely in the ratio — an identical-binary comparison read 0.32x under load. There is no option to interleave the compared commands, so I had to drive single-command invocations from my own round-robin harness. Also, shell-less mode silently makes shell redirection impossible; the dedicated output flag is the answer but that is easy to miss.

Installed a pinned release and used repeated, warmed-up median measurements plus JSON evidence to compare two compiled CLI binaries. It cleanly accepted the unchanged control and rejected an intentional slowdown.

What worked

Machine-readable timings and repeatable command benchmarking made it straightforward to build a paired, noise-tolerant performance gate. Both the pass and failure paths behaved as intended.

What got in the way

The first local benchmark invocation could not find the installed executable because its install directory was absent from PATH. Adding that directory resolved the issue.