Transformer attention concentrates. In practice, trained models develop “attention sinks” — tokens that absorb disproportionate attention weight regardless of semantic relevance — and “massive activations” — individual hidden dimensions with values orders of magnitude larger than their neighbors. These phenomena were discovered empirically and explained as emergent behaviors: complex dynamics of training on language data producing concentration patterns that the architecture doesn't explicitly encode.
The explanation is backwards. The concentration isn't emergent. It's inherited.
The value-softmax structure — the core building block of self-attention, defined as a learnable value matrix times a softmax of attention logits — has a mathematical property: gradient flow on this parameterization inherently drives outputs toward low-entropy solutions. Not because of the data. Not because of the loss function. The property holds across logistic loss, square loss, and other objectives. The softmax-times-matrix structure polarizes under any gradient signal.
Reader note (2026-11-17): Checked against the full text of arXiv:2603.06248 (Varre, Rofin and Flammarion; v1 of March 2026). The theorem matches: gradient flow on the value-softmax parameterization drives outputs toward low entropy across losses. Scope the essay states more strongly than the paper: the model is described by the paper as a stylized simplification of a self-attention block (its derivation ignores the value matrix's effect on other tokens and assumes independence), the analysis uses assumptions such as zero initialization of V and strictly ordered scores, and the paper lists a limitation that the gradient is aligned with the target direction for the losses considered. It reports that in some of its settings the polarization is typically insufficient to yield one-hot scores while with f(x) = x^2 the dynamics can drive them toward a one-hot limit, and that softmax-free attentions (sigmoid, linear) trained in its experiments did not form an attention sink. The paper's claim about transformers is that attention sinks and massive activations can emerge as a consequence of this implicit bias, which it says helps explain sink formation; its motivating question is whether the sparsification is solely enforced by the task.
The proof works by analyzing the implicit bias of gradient flow on the combined V·σ(a) parameterization. As optimization proceeds, the softmax outputs concentrate probability mass — not because the model learns that concentration is useful, but because the gradient landscape has low-entropy solutions as attractors. The model descends toward concentration the way a ball rolls downhill. The direction is set by the parameterization, not the terrain.
Attention sinks and massive activations are the same phenomenon viewed from different measurement frames. The attention sink is the concentration of the softmax output. The massive activation is the corresponding amplification in the value matrix. Both are consequences of the same gradient flow polarization, not independent emergent behaviors requiring separate explanations.
The structure is real. Attention does concentrate. Activations do spike. But the cause was misattributed. What looked like learned behavior — the model discovering that concentration helps — is actually a mathematical property of the optimization surface. The model doesn't choose to concentrate. The gradient gives it no choice.