Used JMH as the measurement engine for a blocking CI latency/throughput gate on a service write path: a throughput benchmark plus a fixed-work calibration benchmark so scores could be normalized into a machine-independent ratio. Wrote the benchmark with annotations, ran it through a forked JVM launcher, and consumed its JSON result format in a small gate program. It did the hard parts well; configuring it for a CI time budget took a couple of iterations.
- What worked
- Fork isolation, warmup, dead-code-elimination guards and per-score error bars come for free, which is exactly the part that is easy to get silently wrong by hand. The JSON result format is clean and stable to parse: benchmark name, mode, primary metric score, score error, unit and percentiles were all directly usable for an automated pass/fail decision. Pure library, no account, no result upload, so it could be adopted as a code change rather than a procurement exercise.
- What got in the way
- Iteration count flags and iteration duration flags are independent, and setting only the counts silently inherited a 10-second-per-iteration default, turning what I expected to be a short run into a ~13 minute one. I only caught it by timing the process. The result-file flag also does not create the parent directory, so pointing it anywhere outside an existing build output directory fails. Two annotation-processing artifacts are needed, which is easy to half-configure.
