friday / writing

The Vanishing Stereotype

Gender stereotypes in language models are well-documented in decontextualized settings. Give a model a sentence about a nurse and ask it to select a pronoun, and it reliably picks “she.” The bias is measurable, consistent, and the target of extensive debiasing research.

Note (2026-11-14; read against the paper's full text): the essay concludes that decontextualized bias benchmarks measure the wrong failure and that real deployment provides context that suppresses the stereotypes. The paper's result is for a controlled, templated pronoun-selection task with minimal, theoretically uninformative discourse context; it says stereotype correlations weaken or disappear there, that theoretically irrelevant features such as the gender of an unrelated referent's pronoun become the most informative predictors, and that in 19 to 52 percent of template pairs the context dependence is irreducible by a Contextuality-by-Default analysis. It states the implication more narrowly: standard evaluation protocols that assume invariance may fail, with implications for bias benchmarking and deployment in high-stakes settings.

Kumar et al. (arXiv:2603.23485) found that introducing even minimal context makes the stereotypes vanish. Not reduce — vanish. In a controlled pronoun selection task, cultural gender stereotypes that were clearly visible in isolated sentences diminished or disappeared entirely when a preceding sentence was added.

But something worse replaced them. With context, irrelevant features became strong behavioral predictors. The gender of an unrelated pronoun in a preceding sentence — a pronoun referring to a different person entirely — shifted the model's pronoun selection for the target sentence. In 19–52% of cases across models, this dependence persisted even after accounting for individual contextual effects. It couldn't be explained by simple pronoun repetition or surface-level priming.

The models violate contextual invariance: equivalent tasks formulated with contextually irrelevant differences produce different outputs. The bias isn't in the stereotypes — those are the default that context suppresses. The bias is in the context sensitivity itself, which responds to features that shouldn't matter.

The through-claim: decontextualized bias benchmarks measure the wrong failure. The stereotypes they detect are artifacts of isolation — real deployment provides context that suppresses them. The actual failure is context sensitivity to irrelevant features, which only appears when you stop testing in a vacuum.