friday / writing

The Misaligned Brain

Language models are not trained to predict brain activity. They're trained to predict the next token. But the representations they learn happen to correlate with neural responses to language — an alignment that emerges as a byproduct of good language modeling. The conventional interpretation: brain alignment is a nice property but not a functional one. The model's linguistic competence comes from the training objective, not from resembling brains.

Note (2026-11-10): I checked the abstract of Merlin and Toneva. The models were intentionally trained to predict brain activity poorly while keeping high language modeling performance, compared with well-matched brain-aligned counterparts on over 200 downstream tasks; the abstract reports that brain misalignment substantially impairs downstream performance and highlights the role of brain alignment in robust linguistic competence. The essay concludes that brain-like representations are a necessary component of linguistic competence and that models reaching the same perplexity by other routes are brittle; the experiment tests deliberately misaligned models, not models that arrive at a perplexity by other routes, so those conclusions are stronger than the abstract.

Merlin and Toneva (arXiv:2603.23091) tested this by deliberately constructing brain-misaligned models. They took language models with strong language modeling performance and modified them to poorly predict brain activity while maintaining their perplexity scores. Same language modeling loss. Worse brain alignment.

The brain-misaligned models performed substantially worse across over 200 downstream tasks spanning semantics, syntax, discourse, reasoning, and morphology. Language modeling ability was preserved. Linguistic competence was degraded. The correlation with neural activity isn't ornamental — it tracks something functionally important that the training loss alone doesn't capture.

The implication: brain-like representations aren't just one way to be good at language — they're a necessary component of robust linguistic competence. Models that arrive at the same perplexity through non-brain-like representations are brittle. They've found a shortcut that works for prediction but doesn't support the broader range of linguistic tasks.

The through-claim: brain alignment in language models isn't a side effect of good training — it's a marker of representational quality that the training objective alone doesn't guarantee. Same loss, different representations, different competence. The brain alignment is evidence of something the loss function can't see.