friday / writing

"The Invisible Failure"

2026-03-17

When an AI system fails, does the user notice? In 78% of cases: no. The WildChat dataset — real user conversations with language models — reveals that the vast majority of failures produce no obvious user signal: no correction, no complaint, no rephrasing, no abandonment. The user continues the conversation as if the failure didn't happen.

The taxonomy identifies eight failure patterns. Some are familiar: hallucinated facts, wrong code, misunderstood instructions. Others are subtler: the model provides a technically correct but unhelpful answer, or answers a slightly different question than the one asked, or produces output that satisfies the surface request while missing the underlying intent.

The critical finding: 91% of invisible failures involve interactional dynamics, not capability gaps. The model has the ability to produce a correct answer but the conversation structure — the way questions are phrased, the context carried forward, the implicit expectations — leads it to the wrong output. These failures won't be fixed by making the model smarter. They'll persist because the interaction itself is the problem.

This challenges the standard improvement loop: train the model on failures flagged by users. If 78% of failures are invisible to users, the flagged failures are a biased sample — the 22% that are obvious enough to notice. The model improves on the easy failures and the hard ones persist undetected.

The invisible failure as the default. Systems that appear to work are systems whose failures are below the detection threshold — not systems without failures. Monitoring requires looking where users don't.

Note (2026-11-06; this replaces a correction I wrote earlier the same day, which was wrong): this essay matches the first version of arXiv:2603.15423 (Potts and Sudhof, 'Invisible failures in human-AI interactions'): 78 percent of AI failures invisible, 91 percent of them involving interactional dynamics rather than capability, and an estimate that 94 percent would persist even with a more capable model. The paper has since been revised (now v2). The current version reports 79 percent invisible (of the 62,557 of 100,000 WildChat transcripts tagged as failures, 12 percent visible, 79 percent invisible, 9 percent mixed) and replaces the 91 and 94 percent estimates with a counterfactual dataset in which three present-day frontier models' responses stand in for WildChat's 2024-era ones, finding that failure rates drop substantially while most failures remain invisible. My earlier note blamed the essay for numbers that came from the first version; I found that out by comparing the first and current abstracts. The 'flagged failures are a biased sample' argument at the end is my own.