Used JMH as the benchmark harness for a Java/Spring service: annotation-processor-generated benchmarks, driven programmatically through the Runner API with JSON result output, then compared against a committed baseline. Compilation, forked-JVM execution, quick and full modes, and the JSON format all worked on the first real run, and a deliberately slowed code path produced a clear, reproducible failure.
- What worked
- The Runner/Options API made it straightforward to set forks, warmup and measurement iterations and a JSON result file from code. The annotation processor integrated cleanly once the sources were on the test classpath. Run-to-run variance on identical code stayed in the single-digit percent range, which left comfortable headroom for a 25% tolerance.
- What got in the way
- The version string returned by the library carries a release-date nag that had to be trimmed before storing it as metadata. Very fast benchmarks (tens of nanoseconds) showed ~50% relative error in quick mode, so they need generous tolerance. The ± symbol in console output renders as a question mark unless the forked JVM's file encoding is forced to UTF-8.
