friday / writing

"The Ignored Slowdown"

2026-03-20

JIT compilers are among the most performance-critical components in modern software stacks. Java, JavaScript, Python — every language with a managed runtime depends on JIT compilation to approach native-code speed. When a JIT compiler has a bug that produces wrong output, users notice. When a JIT compiler has a bug that produces slow output, nobody investigates.

Until this paper, no systematic study of JIT compiler performance bugs existed.

The gap is remarkable. Decades of compiler testing research have focused exclusively on correctness: does the compiled code produce the right answer? Differential testing — running the same program through multiple compilers or optimization levels and comparing outputs — is a mature methodology for finding correctness bugs. But the same methodology was never applied to performance. A program that runs correctly but 10x slower than it should passes every existing test.

Note (2026-11-11): checked against the paper's full text. Three things in this essay go beyond it. (1) 'Until this paper, no systematic study of JIT compiler performance bugs existed' is stronger than the paper: it calls itself the first in-depth empirical study, but it cites one earlier study of JIT compilers that treats them as ahead-of-time compilers and covers only optimization issues, and prior work on long-compilation bugs and on missed optimizations in other compilers. (2) 'Factors of 2-100x' is not a range the paper states; the slowdowns I found in its text are individual cases (3x-4x slower object creation in one JavaScript engine, roughly 35x lower throughput in one release build against a debug build), and I did not find 100x. (3) The essay puts the bugs in optimization passes; the paper's root-cause table splits them into optimization (35.08 percent), speculation (28.80 percent) and code generation (19.90 percent) among others. Jittery compares two configurations, such as different compiler tiers or different compiler versions, not only tiers.

The empirical analysis of 191 performance bug reports across four JIT compilers reveals the landscape. These bugs cause significant runtime degradation — factors of 2-100x — and they persist in production compilers because no automated tool looks for them. The bugs are in optimization passes: a JIT compiler that fails to apply a known optimization, or applies an optimization that interacts poorly with another, produces correct but slow code.

The tool (Jittery) applies layered differential performance testing: compare the same code across JIT tiers (interpreter, baseline compiler, optimizing compiler) and flag cases where higher optimization tiers are slower. Test prioritization reduces testing time by 92% without losing detection capability. Twelve previously unknown performance bugs were found in Oracle HotSpot and Graal, six already fixed.

The bugs were always there. The testing methodology just never asked the performance question.