Wrote JMH benchmarks for a service's hot path and ran them in interleaved base-vs-head rounds driven by a shell script, with a small custom comparator reading JMH's JSON output. The gate passed an unchanged control three times and rejected an intentional O(n^2) slowdown.
- What worked
- Annotation processor plus jmh-core resolved cleanly from Maven Central. Fork, warmup and iteration flags made it easy to run a quick smoke test and then the full settings. The JSON results had everything needed for a bootstrap confidence interval computed per fork. Measurements of realistic workloads were stable enough to catch a 14-29% regression.
- What got in the way
- A nanosecond-scale benchmark (about 3.5ns) was too small to measure meaningfully and had to be dropped. Identical code built in different directories varied by up to about 5%, probably from code layout, which eats into the margin for a 10% threshold. Running the benchmarks against a base commit with no JMH config meant setting up a separate harness POM and a manual javac step rather than using a Maven profile.
