# JMH: Benchmarks That Mean Something

> Ten calls with one returned measured 2.877 ns. The same ten, each consumed, measured 30.721. Nine were never executed, and nothing in the output said so.

- Published: 2026-10-25
- Tags: benchmarking, performance, jit
- Source: https://jvmscope.com/blog/jmh-benchmarks-that-mean-something/
- Language: en-GB
- Author: Marcus Venn

---

## TL;DR

- Ten calls to Math.log with only the last returned measured 2.877 ns/op; the same ten, each consumed by a Blackhole, measured 30.721. Nine calls were eliminated.
- Dropping the result entirely measured 0.302 ns/op against 2.787 for the same call returned - a 9.2x understatement with no warning anywhere in the output.
- An expression over static final ints measured 0.273 ns/op, level with a benchmark that computes nothing, while the same expression over state fields cost 0.417.
- On JDK 26 the benchmark jar built cleanly and contained no benchmarks, because JDK 23 changed javac's default annotation processing policy.

---

Ten calls to `Math.log`, with only the last one returned: 2.877 ns/op. The same ten, each handed to a `Blackhole`: 30.721 ns/op.

Nine of the ten calls were never executed. The first benchmark did not fail, warn, or produce a number that looks implausible — it produced a smaller one, in output identical in shape to the correct measurement beside it. That is the failure mode worth learning, because a benchmark that measures nothing looks exactly like a benchmark that measures something fast.

All figures below: JDK 26.0.2.1 (Homebrew build), macOS 26.6.2, Apple M4 Pro, JMH 1.37, five forks with five warmup and five measurement iterations each — 25 measurements per row.

## The lie has a size

One computation, `Math.log` of a field the compiler cannot fold, wrapped six ways:

| Harness | ns/op |
|---|---|
| `dropped` — computed, not used | 0.302 ± 0.018 |
| `returned` — result returned | 2.787 ± 0.006 |
| `blackholed` — result consumed | 3.236 ± 0.142 |
| `loopReturnsLast` — ten calls, last returned | 2.877 ± 0.030 |
| `loopBlackholed` — ten calls, each consumed | 30.721 ± 1.797 |

`dropped` understates by 9.2 times. That one is well known and easy to avoid: [JMH's own sample](https://github.com/openjdk/jmh/blob/master/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_08_DeadCode.java) states it plainly — "compilers are smart enough to deduce some computations are redundant and eliminate them completely".

The loop pair is the one that survives into published benchmarks, because it follows the advice. It returns its result. It is still 10.7 times wrong, because returning a value protects that value and nothing else. The arithmetic confirms what happened: 30.721 ± 1.797 divided by ten is 3.07 ± 0.18, which overlaps the cost of a single consumed call at 3.236 ± 0.142. The correct benchmark did ten of them. The incorrect one did one.

## The demonstration that did not fold

Constant folding is the second classic, and the textbook example of it did not reproduce.

`Math.log` of a `static final double` measured 3.254 ± 0.150 ns/op against 3.236 ± 0.142 for the same call on a mutable field: the same number, inside each other's error bars. Nothing folded.

The pitfall itself is real, and integer arithmetic shows it immediately:

| Harness | ns/op |
|---|---|
| `foldedArithmetic` — `CA * CB + (CA ^ CB)`, both `static final int` | 0.273 ± 0.012 |
| `liveArithmetic` — identical expression over state fields | 0.417 ± 0.008 |

0.273 is level with `dropped` at 0.302 — the floor, where the harness measures its own loop and nothing else. The expression was evaluated at compile time and the benchmark timed a constant being returned.

Why one folded and the other did not is not something this run establishes, and guessing at it would be the same error the article is about. What it does establish is the practical rule: a benchmark whose score sits at the harness floor has not measured the code, whatever the code looks like.

## The failure before the benchmark

The most complete way for a JMH benchmark to measure nothing is for it not to exist.

On JDK 26 the project compiled, the shade plugin produced `benchmarks.jar`, Maven reported success, and running it produced:

```
Exception in thread "main" java.lang.RuntimeException:
  ERROR: Unable to find the resource: /META-INF/BenchmarkList
```

JMH generates that resource with an annotation processor, and since JDK 23 javac does not run annotation processors by default. The change is documented: "If neither `-processor`, `--processor-path`, nor `--processor-module-path` is used, `-proc:only` or `-proc:full` has to be provided." A `provided`-scope dependency on `jmh-generator-annprocess` puts the processor on the classpath and satisfies none of those conditions, so the generator silently does not run.

The fix is one `annotationProcessorPaths` block in `maven-compiler-plugin`, which sets `--processor-path`. The point is not the fix. It is that every stage of the build reported success, and the only thing that failed was the part that makes the benchmarks exist.

## One fork is one sample

A fork is a fresh JVM, with its own class loading order, its own profile and its own compilation history. The five forks behind each row above did not agree with each other, and how much they disagreed depended on the benchmark:

| Harness | slowest fork | fastest fork | spread |
|---|---|---|---|
| `returned` | 2.794 | 2.783 | 1.00x |
| `loopReturnsLast` | 2.938 | 2.838 | 1.04x |
| `blackholed` | 3.464 | 2.992 | 1.16x |
| `loopBlackholed` | 33.782 | 28.279 | 1.19x |

Same code, same machine, same minute. A single-fork run of `loopBlackholed` would have reported anything from 28.3 to 33.8 ns/op and presented it as the score, and nothing in that run would have hinted at the other 19 per cent. The forks are not noise to average away; they are the sample, and one of them is not a measurement.

The table also prices the `Blackhole`. Consuming a value cost 3.236 against 2.787 for returning it — roughly 0.45 ns of harness. When a benchmark produces one value, return it: it is cheaper and it is stable across forks. The `Blackhole` earns its overhead when there is more than one value, which is precisely the case where returning is not enough.

## Reading a score as a claim

The measurements in [the Optional article](/blog/the-cost-of-optional/) and [the streams article](/blog/streams-versus-loops-what-the-jit-sees/) rest on this, so the method is worth stating explicitly.

A JMH score is a claim about a harness, not a measurement of a method. Three checks make it a measurement. Compare the score against the harness floor: a result near 0.3 ns/op on this machine is the loop, not the code. Consume every value the benchmark computes, not just the one it returns. And read the error bar as a verdict rather than decoration — a run in this plan's own experiments reported 991.098 ± 642.888 ns/op, a 65 per cent interval, in output otherwise indistinguishable from a usable one.

None of that makes a benchmark correct. It makes a wrong one visible, which is the most any harness can offer.

---

## Frequently asked

### Is returning the result enough to prevent dead-code elimination?

Only for the result you return. In the measurement below a loop of ten calls returning the last one measured 2.877 ns/op against 30.721 for the same loop with every value consumed. The nine intermediate results were eliminated, and the benchmark reported a number for work it never performed.

### Why does my JMH jar say "Unable to find the resource - META-INF/BenchmarkList"?

Because the annotation processor that generates it never ran. Since JDK 23, javac requires one of -processor, --processor-path, --processor-module-path, -proc:only or -proc:full before it will run annotation processing at all. Without one the build succeeds and produces a jar with no benchmarks in it.