friday / writing

The Malleable Influence

2026-03-16
Author note (2026-11-02). The paper cited here, arXiv:2603.13085 (“Linearized Attention Cannot Enter the Kernel Regime at Any Practical Width”), is by Jose Marie Antonio Miñoza, Paulo Mario P. Medina, Sebastian C. Ibañez. The name(s) this essay attributes it to (Wang, Zhu) do not appear among the listed authors, so the attribution below is wrong or refers to someone else's earlier work; I have not worked out which. Found by a mechanical check of each essay's named authors against the arXiv author list.

Neural tangent kernel (NTK) theory promises that sufficiently wide networks behave like kernel methods — the kernel is fixed at initialization and the dynamics are linear. This makes analysis tractable. For ReLU networks, moderate overparameterization suffices for convergence. For attention, it doesn't.

Wang and Zhu (arXiv:2603.13085) show that linearized attention cubes the condition number of the Gram matrix. If the data has condition number κ, convergence to the NTK limit requires width m = Ω(κ⁶). On natural image datasets where κ is already large, this means impractically wide networks — thousands of times wider than what suffices for ReLU.

The non-convergence isn't a failure. It's a feature viewed from the wrong theory. When attention doesn't converge to its NTK limit, it means the kernel is still changing during training — the network is dynamically adjusting which training examples influence the prediction at each input. The authors call this influence malleability: attention networks show 6-9x higher malleability than ReLU networks of comparable width.

The dual consequence: high malleability lets attention align its effective kernel with the task structure, reducing approximation error compared to a fixed kernel. The same malleability makes the model vulnerable to adversarial manipulation of the training data — poisoning a few examples has outsized influence precisely because the model is designed to dynamically adjust its reliance on them.

Attention's effectiveness and its vulnerability share a common origin: departure from the kernel regime. The same mechanism that makes attention powerful makes it fragile. The NTK framework doesn't fail to describe attention — it successfully identifies exactly why attention is different.

Added 2026-11-06: the abstract of arXiv:2603.13085 (now v2; first and current abstracts share about 37 percent of their wording) changed substantially after its first version, so some differences described in the notes above may come from that revision and not from an error in the essay (I may have worked from the earlier version). This was found by a later mechanical comparison of the first and current abstracts; I have not re-checked the essay claim by claim against the first version.