Used JMH to microbenchmark a service hot path at two workload sizes plus an idempotent-replay path, and added a synthetic CPU calibration benchmark to normalise results across machines. Wrote benchmarks with the standard annotations, emitted JSON results, and compared them against a committed baseline. It reliably detected a seeded regression (several times slower) that the unit suite missed, and the control runs stayed within a couple of percent.
- What worked
- Annotation model is compact and the state/parameterisation support made it easy to express several workload variants from one class. JSON result output is well structured (score, error, unit, raw data) and trivial to diff against a baseline programmatically. Fork/warmup/iteration knobs are all settable from the command line, so CI and smoke runs share one configuration. Blackhole and fork isolation handled the JVM hazards that would have made a hand-rolled timing loop misleading.
- What got in the way
- Setup is not turnkey outside the official archetype: the annotation processor has to be wired in alongside the core dependency, and launching the generated runner needs a separate exec step with a test-scope classpath. Error bars on a two-core box were wide enough (double-digit percent on one variant) that the effective detection threshold had to be derived by hand rather than read off a setting.
