JMH: Benchmarks That Mean Something

Ten calls to Math.log, with only the last one returned: 2.877 ns/op. The same ten, each handed to a Blackhole: 30.721 ns/op.

Nine of the ten calls were never executed. The first benchmark did not fail, warn, or produce a number that looks implausible — it produced a smaller one, in output identical in shape to the correct measurement beside it. That is the failure mode worth learning, because a benchmark that measures nothing looks exactly like a benchmark that measures something fast.

All figures below: JDK 26.0.2.1 (Homebrew build), macOS 26.6.2, Apple M4 Pro, JMH 1.37, five forks with five warmup and five measurement iterations each — 25 measurements per row.

The lie has a size

One computation, Math.log of a field the compiler cannot fold, wrapped six ways:

dropped understates by 9.2 times. That one is well known and easy to avoid: JMH’s own sample states it plainly — “compilers are smart enough to deduce some computations are redundant and eliminate them completely”.

The loop pair is the one that survives into published benchmarks, because it follows the advice. It returns its result. It is still 10.7 times wrong, because returning a value protects that value and nothing else. The arithmetic confirms what happened: 30.721 ± 1.797 divided by ten is 3.07 ± 0.18, which overlaps the cost of a single consumed call at 3.236 ± 0.142. The correct benchmark did ten of them. The incorrect one did one.

The demonstration that did not fold

Constant folding is the second classic, and the textbook example of it did not reproduce.

Math.log of a static final double measured 3.254 ± 0.150 ns/op against 3.236 ± 0.142 for the same call on a mutable field: the same number, inside each other’s error bars. Nothing folded.

The pitfall itself is real, and integer arithmetic shows it immediately:

0.273 is level with dropped at 0.302 — the floor, where the harness measures its own loop and nothing else. The expression was evaluated at compile time and the benchmark timed a constant being returned.

Why one folded and the other did not is not something this run establishes, and guessing at it would be the same error the article is about. What it does establish is the practical rule: a benchmark whose score sits at the harness floor has not measured the code, whatever the code looks like.

The failure before the benchmark

The most complete way for a JMH benchmark to measure nothing is for it not to exist.

On JDK 26 the project compiled, the shade plugin produced benchmarks.jar, Maven reported success, and running it produced:

Exception in thread "main" java.lang.RuntimeException:
  ERROR: Unable to find the resource: /META-INF/BenchmarkList

JMH generates that resource with an annotation processor, and since JDK 23 javac does not run annotation processors by default. The change is documented: “If neither -processor, --processor-path, nor --processor-module-path is used, -proc:only or -proc:full has to be provided.” A provided-scope dependency on jmh-generator-annprocess puts the processor on the classpath and satisfies none of those conditions, so the generator silently does not run.

The fix is one annotationProcessorPaths block in maven-compiler-plugin, which sets --processor-path. The point is not the fix. It is that every stage of the build reported success, and the only thing that failed was the part that makes the benchmarks exist.

One fork is one sample

A fork is a fresh JVM, with its own class loading order, its own profile and its own compilation history. The five forks behind each row above did not agree with each other, and how much they disagreed depended on the benchmark:

Same code, same machine, same minute. A single-fork run of loopBlackholed would have reported anything from 28.3 to 33.8 ns/op and presented it as the score, and nothing in that run would have hinted at the other 19 per cent. The forks are not noise to average away; they are the sample, and one of them is not a measurement.

The table also prices the Blackhole. Consuming a value cost 3.236 against 2.787 for returning it — roughly 0.45 ns of harness. When a benchmark produces one value, return it: it is cheaper and it is stable across forks. The Blackhole earns its overhead when there is more than one value, which is precisely the case where returning is not enough.

Reading a score as a claim

The measurements in the Optional article and the streams article rest on this, so the method is worth stating explicitly.

A JMH score is a claim about a harness, not a measurement of a method. Three checks make it a measurement. Compare the score against the harness floor: a result near 0.3 ns/op on this machine is the loop, not the code. Consume every value the benchmark computes, not just the one it returns. And read the error bar as a verdict rather than decoration — a run in this plan’s own experiments reported 991.098 ± 642.888 ns/op, a 65 per cent interval, in output otherwise indistinguishable from a usable one.

None of that makes a benchmark correct. It makes a wrong one visible, which is the most any harness can offer.