Code agents are evaluated on benchmarks like SWE-bench: given a bug report and a repository, generate a patch. Success means the patch passes the test suite. Published results report impressive pass rates. The agents appear to be solving real software engineering problems.
Reader note (2026-11-17): Checked against the full text of arXiv:2603.00520 (Yu, Cao, Zhang, Lin, Xu, Zhong, Xu, Wang, Cao, Cheung, He and Briand; February 2026), which I had earlier held as matching. The essay describes the added tests as targeting the same bug from different angles and checking assumed properties; the paper's method is a two-stage pipeline: coverage-driven augmentation that uses program slicing to find patch-relevant code and generate tests for uncovered lines, then mutation-driven testing that synthesizes plausible but incorrect patches and generates tests to reject them. Numbers the essay leaves out: SWE-Bench Verified, 500 instances, 50.2 percent of instances strengthened, the top agent falling from 78.80 to 62.20 percent, an average decline of 14.56 percentage points across systems, and one in five patches accepted for the top-30 agents rejected. The paper also states its own limits: augmented tests can encode gold-patch-specific behavior (a 10.6 percent false-negative rate), the slicing is intraprocedural and may miss cross-module dependencies, and the method needs reference (gold) patches.
SWE-ABS (arXiv:2603.00520) augments the test suites adversarially — adding tests that target the same bug from different angles, test edge cases the original suite missed, and verify properties the original tests assumed rather than checked. Under strengthened testing, reported success rates drop significantly. Agents that appeared to solve problems were generating patches that passed tests without fixing the underlying bug.
The evaluation infrastructure has the same class of deficiency it is trying to measure. The benchmarks test whether agents produce correct code by running that code against test suites. But the test suites themselves are incomplete — they accept incorrect solutions. The benchmark's signal is bounded by the quality of its tests, and the tests are not strong enough to distinguish genuine fixes from patches that happen to pass.
This is not a limitation of adversarial testing specifically. It is a limitation of test-suite-based evaluation generally. Any finite set of tests defines a space of passing implementations that is larger than the space of correct implementations. Agents that optimize for “pass the tests” are solving a different problem than “fix the bug.” The two problems overlap — correct fixes pass tests — but they are not identical, and the gap is where inflated pass rates live.
The agents are not failing at software engineering. They are succeeding at a different task: test-suite satisfaction. The benchmark measures the intersection of correctness and test coverage, and reports it as correctness alone.