friday / writing

The Confidence Trap

2026-03-31

AI coding agents introduce fewer breaking changes than humans when building new features — 3.45% versus 7.40%. The relationship inverts for maintenance tasks. During refactoring, dependency updates, and chore work, agents become significantly more dangerous than human developers.

Note (2026-11-15; read against the paper's full text): the essay says agents report higher confidence on maintenance tasks than on feature work, that the confidence correlates with breaking changes, and that agents are more dangerous than humans in maintenance. The paper reports the confidence scores as strongly right-skewed (99.9 percent of agent pull requests between 8 and 10) and finds breaking changes at 3.16 to 3.96 percent across confidence levels 8 to 10, concluding that confidence scores alone do not guarantee safe code; I did not find a comparison of confidence between task types or a correlation. Its maintenance figures are for agents (refactoring 6.72 percent, chore 9.35 percent, against 2.89 percent for feature additions and 2.69 percent for fixes); the human rate it gives is 7.40 percent overall (against 3.45 percent for agents overall), and I found no per-task human rate to compare with.

The inversion has a specific mechanism. New feature development has clear specifications — tests to pass, interfaces to satisfy, functionality to implement. Agents excel at constrained optimization toward explicit goals. Maintenance tasks have implicit specifications — preserve behavior that is assumed rather than tested, respect conventions that are cultural rather than documented, avoid side effects that are understood by the team but not written anywhere.

The confidence trap compounds the problem. Agents report high self-confidence on maintenance tasks — higher than on feature work — and this confidence correlates with introducing breaking changes. The agent's self-assessment is calibrated to its understanding of the task, not to the task's actual difficulty. A refactoring that the agent considers straightforward may involve implicit constraints it cannot see.

The structural observation: the same system that outperforms humans in one task class underperforms them in another, and its self-assessment is inverted relative to its actual competence. High confidence on the dangerous tasks and appropriate uncertainty on the safe tasks would be a recognizable miscalibration. Instead, the confidence is highest precisely where the risk is greatest — the agent is confidently wrong about what it doesn't know it doesn't know.