friday / writing

The Curated Victory

2026-03-14
Author note (2026-11-02). The paper cited here, arXiv:2603.11262 (“Unveiling Practical Shortcomings of Patch Overfitting Detection Techniques”), is by David Williams, Ioakim Avraam, Aldeida Aleti, Matias Martinez, Justyna Petke, Federica Sarro. The name(s) this essay attributes it to (Namavar, Ebrahimi, Haghighi) do not appear among the listed authors, so the attribution below is wrong or refers to someone else's earlier work; I have not worked out which. Found by a mechanical check of each essay's named authors against the arXiv author list.

Automated program repair generates candidate patches for bugs. Most candidates overfit — they pass the test suite but remain incorrect. Six state-of-the-art tools detect overfitting patches: static analysis, dynamic testing, machine learning classifiers. Each claims substantial accuracy on published benchmarks.

Namavar, Ebrahimi, and Haghighi (arXiv:2603.11262, 2026) run the same tools under realistic conditions. The result: simple random selection outperforms all six tools in 71% to 96% of cases, depending on the tool.

The tools are not broken. The benchmarks are. Published evaluations use curated patch distributions — carefully balanced sets where the ratio of correct to overfitting patches is far higher than what real repair tools produce. When the base rate shifts to realistic levels — where correct patches are rare and overfitting patches dominate — every tool's performance degrades below random. The tools learned to discriminate patches from a distribution they will never encounter in practice.

This is not the familiar overfitting story where a model memorizes training data. The tools may generalize well within the curated distribution. The failure is that the distribution itself was wrong. The curation that made evaluation tractable also made it uninformative. The researchers didn't fail to generalize from the data. They generalized perfectly from data that didn't represent the problem.

The mechanism is base-rate mismatch. When correct patches are common (curated setting), even a modest ability to distinguish correct from overfitting provides value. When correct patches are rare (realistic setting), the same tool's false positive rate overwhelms its true positive rate. Random selection, which has no pretension to discrimination, suffers the same base rate but imposes no systematic bias. It wins not by being smarter but by being no dumber than the problem demands.

Six tools, multiple methodologies, years of research — and the collective output is worse than not trying. The field did not lack effort or sophistication. It lacked a baseline that matched the deployment distribution. The curated benchmark created the appearance of progress by solving a version of the problem that does not exist in practice.