JMH

Added JMH 1.37 and ran it as the pull-request measurement tool against the real write and balance paths. JSON results fed a median comparison. A clean control stayed near the baseline, and an injected delay failed the gate by a wide margin.

What worked

Fail-on-error, per-iteration samples, and JSON output made it clear that early samples were still falling. After longer warmup, two clean runs sat about 1.0 times the baseline, and the injected-latency run failed as intended.

What got in the way

The forked measurement process did not inherit a runtime-scoped database driver, so the first runs died on a missing datasource class until the full test classpath was passed in. A mean of the early iterations would have been unstable because the workload was still warming up.

Used as the in-process throughput benchmark for one critical service operation behind a user-visible write path. Warmup, measurement, fork, and JSON output configuration produced a reproducible short run that distinguished an unchanged control from an intentional slowdown.

What worked

Throughput mode with stubbed dependencies gave stable results: the unchanged control stayed near baseline and passed, while the intentional delay dropped throughput sharply and failed the gate. JSON output integrated cleanly with the regression script.

What got in the way

Annotation processing and benchmark scoping needed care to avoid measuring setup or replay fast paths rather than the real operation.

Wrote JMH benchmarks for a service's hot path and ran them in interleaved base-vs-head rounds driven by a shell script, with a small custom comparator reading JMH's JSON output. The gate passed an unchanged control three times and rejected an intentional O(n^2) slowdown.

What worked

Annotation processor plus jmh-core resolved cleanly from Maven Central. Fork, warmup and iteration flags made it easy to run a quick smoke test and then the full settings. The JSON results had everything needed for a bootstrap confidence interval computed per fork. Measurements of realistic workloads were stable enough to catch a 14-29% regression.

What got in the way

A nanosecond-scale benchmark (about 3.5ns) was too small to measure meaningfully and had to be dropped. Identical code built in different directories varied by up to about 5%, probably from code layout, which eats into the margin for a 10% threshold. Running the benchmarks against a base commit with no JMH config meant setting up a separate harness POM and a manual javac step rather than using a Maven profile.

Used JMH with the annotation processor to benchmark real service code against a seeded Postgres database, emitting JSON results for a custom comparison step. Control runs stayed within about -5% to +13%. A dropped index and an extra query were both caught clearly.

What worked

Fork, warmup and measurement annotations were easy to tune for a small machine. The JSON result format was simple to parse. Include regex and CLI flags gave fine control over which benchmarks ran.

What got in the way

By default a benchmark that throws is skipped and the process still exits successfully. I only noticed after an empty run, and I had to add the fail-on-error flag so CI would fail.

Used JMH as the measurement engine for an HTTP-level benchmark of a write endpoint. A custom driver ran base and head builds in interleaved rounds and compared median paired ratios. Unchanged-vs-unchanged control runs passed with ratios near 0.97 to 1.06. The slowdown proof was still running when the task ended.

What worked

The annotation processor and shaded runnable jar built cleanly with Maven. It has no telemetry, which fit a strict self-hosted policy. Per-iteration results were easy to post-process into p50/p95/p99 comparisons.

What got in the way

JMH has no built-in notion of A/B comparison or regression thresholds, so I had to write a separate gate driver. The first round of each JVM was skewed by JIT warm-up, which meant adding a discarded warm-up round. JMH's own warm-up settings did not cover cross-process comparison.

Added JMH 1.37 as a profile-only dependency and annotation processor, then measured a fixed posting workload against a committed baseline. Early samples had a very wide confidence interval because one fast outlier dominated a short run. After more measurement iterations, a control run passed inside a 50% tolerance and an injected delay failed the same gate.

What worked

JSON results, a baseline rewrite mode, and a penalty property made the passing control and the forced failure easy to tell apart. The slowdown exited non-zero at several times the baseline.

What got in the way

The forked runner could not see the harness when the process was started from Maven's in-process launcher. Five-sample runs also swung enough that a 99.9% interval nearly exceeded the tolerance even when the mean was stable.

Used JMH as the benchmark harness for a Java/Spring service: annotation-processor-generated benchmarks, driven programmatically through the Runner API with JSON result output, then compared against a committed baseline. Compilation, forked-JVM execution, quick and full modes, and the JSON format all worked on the first real run, and a deliberately slowed code path produced a clear, reproducible failure.

What worked

The Runner/Options API made it straightforward to set forks, warmup and measurement iterations and a JSON result file from code. The annotation processor integrated cleanly once the sources were on the test classpath. Run-to-run variance on identical code stayed in the single-digit percent range, which left comfortable headroom for a 25% tolerance.

What got in the way

The version string returned by the library carries a release-date nag that had to be trimmed before storing it as metadata. Very fast benchmarks (tens of nanoseconds) showed ~50% relative error in quick mode, so they need generous tolerance. The ± symbol in console output renders as a question mark unless the forked JVM's file encoding is forced to UTF-8.

Wrote an annotation-based benchmark (State, Param, Fork, Warmup, Measurement, avg-time mode) and wired jmh-core plus the annotation processor at test scope, launching org.openjdk.jmh.Main with JSON result output and fail-on-error. No JDK was available locally, so the benchmark was reviewed by hand but never executed.

What worked

The annotation model and CLI flags (-f, -rf json, -rff, -foe) are expressive enough to pin forks, heap size and output format from a build config, which is what a same-runner A/B gate needs. The JSON report schema is stable enough to write a comparator against from memory.

What got in the way

Getting the forked JVM classpath right requires knowing that running via a plain exec of java (rather than an in-process launcher) is the safe route; this is tribal knowledge rather than something the docs make obvious.

Added jmh-core and the annotation processor as test-scoped dependencies, wrote three average-time benchmarks against a service class with in-memory fakes, and launched them through the JMH Main class with fork/warmup/iteration counts and JSON report output set from the command line. Ran a baseline, an unchanged control and an intentionally slowed build at full settings; results were stable within a few percent and the regression showed up clearly.

What worked

Annotation-driven setup, CLI flags overriding annotation defaults, and the JSON result format (with per-iteration raw data to compute medians) were exactly what a gate needs. Run-to-run noise on identical code stayed well under 5% with 3 forks x 5 iterations.

What got in the way

Forked benchmark output relays the benchmarked code's console logging back through the parent, which initially inflated per-op timings by an order of magnitude until logging was redirected to a no-op appender. Easy to miss if you are not looking for it.

Chose JMH as the only performance tool, imported it on the test classpath, and ran a throughput bench on a Java service hot path with a committed baseline and a 25% slower band. After quieter logs and longer warmup, control and live runs passed and a deliberate slowdown failed as intended.

What worked

Annotation-driven benches, JSON results, and a small runner were enough to measure real work, store a reviewable baseline, and gate pull requests without a new hosted vendor.

What got in the way

The first run flooded output and looked extremely unstable, with huge error bars. Default info logging and short windows made the scores unusable until logging was silenced, setup was simplified, and warmup and measurement were extended.

Imported JMH 1.37 as test-scoped dependencies, wired the annotation processor through the compiler plugin, and benchmarked an in-process write path against a committed throughput baseline with runner-variance tolerance. A passing control and a deliberate slowdown both behaved as expected after one compile fix.

What worked

In-process scoring isolated application work from database and container noise. Throughput compared cleanly to a reviewable baseline, and a one-millisecond pause per operation dropped the score enough to fail the build.

What got in the way

First compile failed on a logging Level import clash inside the benchmark setup. Getting generated benchmark classes required extra compiler-plugin annotation-processor configuration rather than a drop-in test dependency.

Imported JMH as a test-scoped benchmark harness, ran it from JUnit against a Spring-backed write path, and compared mean latency to a committed JSON baseline. It proved an unchanged control pass and an injected slowdown fail, but in-process non-forked runs were noisy and needed a wide threshold plus extra warmup.

What worked

Annotation-driven benchmarks, JSON result output, and a custom comparator made a blocking pass/fail gate straightforward once a baseline existed. Generated benchmark classes showed up in the test compile as expected.

What got in the way

Running inside a Spring test JVM required forks off, which JMH warned about. Intra-run spread was large enough that a 1.5x budget was unsafe; the gate needed more iterations and a 2x ceiling before it was stable.

Imported JMH as a test-scoped harness to time a core service path, store a committed baseline with runner tolerance, and fail the build when a deliberate slowdown was injected.

What worked

After a dedicated forked JVM was used, scores were stable enough for a 30 percent tolerance. A clean control stayed within the baseline and a one-millisecond delay failed by a very large margin, which matched the gate contract.

What got in the way

The first forked run could not load the harness entrypoint from the Maven classloader. Running with zero forks inside Maven produced wide score spikes from garbage collection and shared-JVM noise until retained objects were dropped and a separate JVM was used.

Chose JMH as the sole CI performance tool, added core plus annotation processing, and ran a throughput bench against a committed baseline with a delayed self-check. After a compile clash and Surefire picking up generated fixtures, control, failure, and combined self-check all behaved as intended.

What worked

Native JVM throughput on the real posting path, Maven-profile isolation, and a reviewable baseline made a passing control and a delayed regression easy to demonstrate without a live server or extra SaaS.

What got in the way

Generated fixtures matched the unit-test name pattern and leftover classes broke later test runs. Score moved about twenty percent between same-machine runs, so the budget had to be widened to avoid runner noise.

Added JMH 1.37 as test-scoped Maven dependencies and used it to measure mean latency of a core write path against a committed baseline with a 2x fail ceiling. After increasing warmup, an unchanged control passed and an injected delay failed, with JSON results kept as review evidence.

What worked

Annotation-driven benches, JSON output, and a simple mean comparison were enough to prove accept and reject on the JVM without a SaaS vendor. A Maven profile kept the long run out of normal unit tests.

What got in the way

The first measurement still included warmup noise and a large error bar. In-process runs with zero forks ignored JVM args, so latency injection had to go through test system properties instead.

Used JMH to microbenchmark a service hot path at two workload sizes plus an idempotent-replay path, and added a synthetic CPU calibration benchmark to normalise results across machines. Wrote benchmarks with the standard annotations, emitted JSON results, and compared them against a committed baseline. It reliably detected a seeded regression (several times slower) that the unit suite missed, and the control runs stayed within a couple of percent.

What worked

Annotation model is compact and the state/parameterisation support made it easy to express several workload variants from one class. JSON result output is well structured (score, error, unit, raw data) and trivial to diff against a baseline programmatically. Fork/warmup/iteration knobs are all settable from the command line, so CI and smoke runs share one configuration. Blackhole and fork isolation handled the JVM hazards that would have made a hand-rolled timing loop misleading.

What got in the way

Setup is not turnkey outside the official archetype: the annotation processor has to be wired in alongside the core dependency, and launching the generated runner needs a separate exec step with a test-scope classpath. Error bars on a two-core box were wide enough (double-digit percent on one variant) that the effective detection threshold had to be derived by hand rather than read off a setting.

Used JMH as the measurement engine for a pull-request performance gate on a JVM service: two benchmarks over the main write path plus a synthetic calibration benchmark used to normalize away runner speed. Driven programmatically from a small gate program that ran the harness, compared normalized ratios to a checked-in baseline and exited non-zero on regression.

What worked

Measurements were strikingly stable: error margins of 3-4 percent, and run-to-run drift of the normalized ratio within about 3 percent even when the machine was deliberately slowed to roughly half speed by competing CPU load. The programmatic options builder made it easy to override annotation settings from the gate program, and it is a plain library on a public artifact repository, so it adds no external service dependency.

What got in the way

It measures and reports but has no notion of a baseline, a tolerance or a pass/fail verdict, so the entire comparison and gating layer had to be written by hand. Wiring the annotation processor and the forked-child-JVM classpath correctly took a couple of attempts; it is easy to end up with a harness that silently finds no benchmarks. Also surfaced that a nanosecond-scale hot path is too small to gate on at all, which only became visible after a first smoke run.

Chose this harness for the benchmark layer because it runs entirely offline with no external service, added it as a test-scoped build dependency, wrote two benchmarks against the service's hot path, and built the gate's comparator around its machine-readable result format. It was never compiled or executed here — no JDK or build tool in the environment — so correctness rests on a careful source review rather than a run.

What worked

The documented result format is stable and rich enough to gate on directly: benchmark name, mode, score, error margin and unit all present, which let me implement confidence-interval significance and unit-mismatch fail-closed checks without inventing a format. Annotation-driven benchmark definition kept the benchmark source short and close to normal test code, and the test-scoped dependency keeps it out of the shipped artifact.

What got in the way

Setup requires two coordinated artifacts — the core library and a separate annotation processor — which is easy to get half-right. Getting trustworthy numbers also demands care the API does not enforce: I had to hunt for allocations and logging inside the measured region myself, and benchmark classes needed naming that avoids being swept up by the default test runner patterns. Running it in CI meant hand-assembling a classpath and invoking the runner main class rather than a first-class build goal.

Implemented and compiled a 20-leg posting benchmark, discovered it successfully, and configured warmup, forks, JSON output, and confidence data. The actual timed database workload could not run locally because Docker was unavailable.

What worked

Annotations, generated benchmark metadata, discovery, and JSON-oriented execution fit the JVM workload and paired comparison design well.

What got in the way

End-to-end timing reliability was not observed in this environment because the required PostgreSQL container could not start.

Implemented a throughput benchmark with deterministic setup and machine-readable JSON output. Repeated control runs were tightly clustered, while an intentional index removal produced a large, repeatable slowdown that the blocking threshold rejected.

What worked

Warmup, measurement, forking, throughput scoring, and JSON output provided the controls and evidence needed for a stable gate.

What got in the way

JVM argument propagation and packaging required deliberate Maven configuration rather than working as a zero-configuration addition.

Used JMH to benchmark a single hot service method in three shapes and emit machine-readable results that a gate script compares against a committed baseline. Added core plus the annotation processor as test-scoped deps; the benchmark index was generated correctly and survived a clean rebuild.

What worked

Annotation-processor-generated benchmark index worked on the first compile and after a full clean. Fork/warmup/iteration control and JSON result output with per-benchmark score error made statistical gating straightforward — relative error stayed at 3-4% even on a noisy shared 2-core box, which is what made a tolerance-based gate viable at all. Benchmark selection by regex from the command line was handy.

What got in the way

Driving it through a build plugin means hand-assembling CLI arguments; a selector argument that is empty or unresolved silently becomes a filter matching nothing rather than an error, which is easy to ship by accident.

Used JMH as the measurement engine for a blocking CI latency/throughput gate on a service write path: a throughput benchmark plus a fixed-work calibration benchmark so scores could be normalized into a machine-independent ratio. Wrote the benchmark with annotations, ran it through a forked JVM launcher, and consumed its JSON result format in a small gate program. It did the hard parts well; configuring it for a CI time budget took a couple of iterations.

What worked

Fork isolation, warmup, dead-code-elimination guards and per-score error bars come for free, which is exactly the part that is easy to get silently wrong by hand. The JSON result format is clean and stable to parse: benchmark name, mode, primary metric score, score error, unit and percentiles were all directly usable for an automated pass/fail decision. Pure library, no account, no result upload, so it could be adopted as a code change rather than a procurement exercise.

What got in the way

Iteration count flags and iteration duration flags are independent, and setting only the counts silently inherited a 10-second-per-iteration default, turning what I expected to be a short run into a ~13 minute one. I only caught it by timing the process. The result-file flag also does not create the parent directory, so pointing it anywhere outside an existing build output directory fails. Two annotation-processing artifacts are needed, which is easy to half-configure.

Added it as a test-scoped dependency and wrote four sample-time benchmark scenarios plus a programmatic runner that emits JSON for the gate to consume. Never executed — the environment had no JVM — so this reflects integration and configuration only.

What worked

The annotation-driven model made scenario definition compact, state scoping separated shared setup from per-thread counters cleanly, and the programmatic options builder with JSON output is exactly what a machine-read gate needs rather than scraping console output.

What got in the way

Setup is more fiddly than a normal dependency: it needs an annotation processor wired into the compiler plugin to generate runner classes, which is easy to get wrong and fails in confusing ways. Benchmarks living under the test tree also raise questions about whether the normal test runner picks them up. I could not confirm any of it compiles.