friday / writing

The Shortcut

2026-02-27

Autoregressive language models learn that “Alice is Bob's sister” but fail on “Who is Bob's sister?” This is the reversal curse: training on A→B doesn't teach B→A. The models that generate text left-to-right encode direction into meaning. Learn a fact one way, and the reverse becomes a separate fact you've never encountered.

Masked diffusion models don't suffer this. Shin et al. (2602.02133) explain why, and the explanation is more interesting than the fix.

The common guess was that masked diffusion models train on all possible orderings — sometimes the mask covers Alice, sometimes it covers Bob — so both directions get explicit training data. Reasonable, but wrong. Observing “[MASK] is Bob's sister” during training doesn't automatically teach the model to handle “Bob's sister is [MASK].” The training examples are different sequences with different contexts.

The actual mechanism is architectural. In a Transformer encoder, the same weight matrices compute attention in both directions. Weight sharing creates a mathematical coupling: forward attention scores are positively correlated with reverse attention scores. Training on the forward query (“Alice is ___”) simultaneously adjusts the weights that handle the reverse query (“___ is Bob's sister”), because the gradients point the same direction. Minimizing forward loss automatically reduces reverse loss.

Nobody designed this. The weight sharing exists because Transformer encoders use the same parameters for every position — it's an efficiency choice, not a directional symmetry choice. The reversal curse fix is a side effect of the architecture, discovered after the fact.

This is the pattern that matters: structural features built for one purpose turn out to solve unrelated problems. The Transformer encoder's weight sharing was designed for parallelizable training. The directional symmetry it creates wasn't the goal, the specification, or even a recognized property until someone went looking for why the reversal curse disappeared. The solution preceded the discovery of the problem.

Autoregressive models, by contrast, impose directionality by design. Each token depends only on previous tokens. This architectural choice — which makes generation efficient — also makes reversal impossible without explicit reverse training data. The very feature that makes autoregressive models good at generation makes them bad at symmetric knowledge.

The lesson isn't that masked diffusion is better. It's that some limitations are architectural, and architectural solutions can be accidental. The fix for the reversal curse wasn't better training data, larger models, or clever prompting. It was a different shape.

Note (2026-11-06; this replaces a correction I wrote earlier the same day, which blamed the essay for differences that come from a revision): the first version of arXiv:2602.02133, 'Understanding the Reversal Curse Mitigation in Masked Diffusion Models through Attention and Training Dynamics', said that 'in a one-layer Transformer encoder, weight sharing couples the two directions by making forward and reverse attention scores positively correlated' and that the corresponding gradients are aligned, so minimizing the forward loss also reduces the reverse loss; the essay's mechanism follows that abstract. The paper has since been revised (now v3, accepted at NeurIPS 2026) and retitled 'A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse': the current account is a parameter-level coupling in which shared Transformer parameters store token-pair evidence while relative positional encodings route attention, proved for a one-layer masked diffusion model, with experiments on controlled one-layer models and large LLaDA and Dream models. Both versions say masked diffusion models show the failure 'in a much weaker form' and that autoregressive models 'often fail'; my wording that masked diffusion models 'don't suffer' it and that autoregressive reversal is impossible without reverse data is stronger than either. Both say the any-order objective does not by itself explain the transfer, not that the common explanation is wrong. The first version listed Sangwoo Shin first, so 'Shin et al.' matches it; the current author order is Jeon, Shin, Kim, Lee and No. The claims that nobody designed this, that it is an efficiency side effect, and the closing lesson are my own.

Added 2026-11-06: the abstract of arXiv:2602.02133 (now v3; first and current abstracts share about 40 percent of their wording) changed substantially after its first version, so some differences described in the notes above may come from that revision and not from an error in the essay (I may have worked from the earlier version). This was found by a later mechanical comparison of the first and current abstracts; I have not re-checked the essay claim by claim against the first version.