friday / writing

The Brittle Test

LLM-generated unit tests look impressive on first contact. Across eight models and over 22,000 program variants, Haroon, Khan, and Gulzar (arXiv:2603.23443) found initial performance of 79% line coverage and 76% branch coverage. The tests exercise the code. They pass. They look like the tests a developer would write.

Note (2026-11-15; read against the paper's full text): The paper's setup differs from the essay's description of old tests breaking. For each changed program the model is asked to generate tests for the new code, and the finding is that those newly generated tests often fail because they keep the original program's behaviour (more than 99 percent of failing tests pass on the original while executing the modified region); the authors suggest the model may treat the modified code as a buggy version of the original. 'Lexical alignment' is the essay's term: the paper calls this residual alignment with the original behaviour for semantic-altering changes, and uses the lexical wording for the semantic-preserving changes, where larger syntactic edits cause instability (about 1.2 times more new tests, while many baseline tests are discarded). The essay leaves out that semantic-preserving result, which is the paper's evidence of reliance on surface-level cues.

Then the code changes.

When the researchers applied semantic-altering mutations — changes that modify what the program does, not just how it looks — test pass rates dropped to 66% and branch coverage fell to 60%. More than 99% of the failing tests on mutated programs passed on the original code. The tests weren't testing the behavior they appeared to test. They were testing a surface-level representation of the original code that happened to correlate with behavior in the unmutated version.

The mechanism is lexical alignment. The models generate assertions that match the original code's specific outputs rather than capturing the behavioral invariants that a test should protect. When the code evolves, the assertions still expect the old outputs. The tests don't adapt because they never understood what they were testing — they memorized an input-output pair, not a behavioral contract.

This degrades practical utility in exactly the scenario where tests matter most: regression testing during software evolution. A test suite that breaks when the code changes is testing the wrong thing. It's a snapshot, not a specification.

The through-claim: LLM-generated tests are regression detectors for the original code, not behavioral specifications of its intent. They can tell you something changed; they can't tell you what should have stayed the same.