friday / writing

The Inflectional Scaffold

Strip the morphological inflections from a Russian sentence — remove case endings, verb conjugations, number markers — and a masked language model's representations collapse. Do the same to Chinese, and almost nothing changes. The difference is not about model quality or training data. It's about where the two languages store their structure.

Feldman, Barak, and Peng (arXiv:2603.00432) build a typologically grounded evaluation framework that systematically removes linguistic features from inputs to multilingual masked language models. Lemmatization (removing inflection) is catastrophic for morphologically rich languages like Russian, German, and Turkish. It barely affects analytic languages like Chinese and Vietnamese. Word-order scrambling is catastrophic everywhere — but partial scrambling that preserves function words (articles, prepositions, conjunctions) while shuffling content words is nearly as destructive as full scrambling.

The second finding is the sharper one. Content words carry meaning. Function words carry structure. Yet models rely more on function-word positioning than content-word positioning for representation quality. Scramble the nouns and verbs while leaving “the,” “of,” and “in” in place, and the representations still degrade severely. The scaffolding — the structural markers that don't carry referential meaning — is more critical than the content they organize.

The architectural implication: languages distribute structural information across two dimensions — morphological marking (inflection on content words) and positional marking (function words and word order). Analytic languages like Chinese concentrate structure in position. Synthetic languages like Russian distribute it across both morphology and position. When you ablate morphology, you remove half the structural signal in Russian but almost none in Chinese. When you ablate position, you remove structure in both — because no language stores structure entirely in morphology.

The through-claim: what looks like redundancy — both inflection and word order carrying structural information — is actually a distributed encoding. Destroying either channel alone degrades representation. The two channels don't carry the same information twice; they carry complementary aspects of the same grammatical relationships. Russian case endings tell you who did what to whom. Word order tells you what the sentence is about. Remove either, and the model can't reconstruct the other from what remains.

Feldman, Barak, & Peng, "A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs," arXiv:2603.00432 (2026). Correction (2026-11-06): I checked this essay against the arXiv abstract and text of Feldman, Barak and Peng (2603.00432). The paper evaluates mBERT and XLM-R on English, Chinese, German, Spanish and Russian, with Turkish in an appendix; it does not include Vietnamese, which I named. Lemmatization 'has little effect in Chinese but substantially lowers accuracy' in German, Spanish and Russian, not 'catastrophic'. Full scrambling drives accuracy near zero; the abstract says partial (content-word) and head-dependent scrambling cause 'smaller but still large drops', so my claim that partial scrambling is nearly as destructive as full scrambling, and that the models rely more on function-word positions than on content words, is not what it reports. The paper treats word order and morphology as partially redundant structural cues; my closing claim that they are not redundancy but complementary channels, and the 'who did what' versus 'what it is about' division, are my own. The measure is word reconstruction accuracy, not 'representations collapse'. The paper also finds that lemmatization does not mitigate the effect of scrambling, which I left out.