Attention is a gradient flow. Each token moves toward other tokens it finds relevant, pulled by the gradient of an interaction energy. The dynamics are first-order: tokens have positions but no momentum. They creep downhill.
Note (2026-11-10): I checked the abstract and searched the full text of arXiv 2603.16535 (Stein, Li, Steidl). The proofs in the paper are narrower than the essay's “provably faster” and “the acceleration is exact”. For linear self-attention it proves that the accelerated flow is a Stein variational gradient flow and that elliptically contoured distributions are preserved; the faster convergence is shown numerically (the abstract says demonstrate), on TinyStories. The essay's “same computational cost” does not match the paper's Table 1: the baseline takes about 2,409 seconds of wall-clock time and the presymplectic variants about 4,979 to 5,798 seconds, roughly twice as long, although the paper says the number of oracle calls is preserved. The loss results are mixed: the last-step validation loss is 2.4687 for the baseline, 2.4728 for the presymplectic Euler variant and 2.6653 for the AB-2 variant (both worse), and lower for the exponential-Euler variant (2.3579) and the ETD-AB-2 variant (1.8386).
Stein, Li, and Steidl give tokens velocity. In their SympFormer architecture, each token carries two variables: a spatial feature vector (where it is in representation space) and a velocity vector (how fast and in what direction it is moving). The resulting dynamics are Hamiltonian — second-order, momentum-carrying, symplectically structured.
This is Nesterov acceleration applied not to parameter optimization but to the attention mechanism itself. Classical attention is gradient descent on the interaction energy between tokens. SympFormer is Nesterov's accelerated method on the same energy, but formulated on the space of probability densities equipped with Wasserstein-2 geometry. The acceleration is exact: the Hamiltonian structure preserves elliptically contoured distributions and converges provably faster than first-order attention, without additional oracle calls.
The practical consequence: tokens with momentum overshoot, correct, and settle faster than tokens that merely creep. A token moving toward a cluster of relevant neighbors carries its velocity past the first neighbor, arriving at the cluster center in fewer steps. The momentum is not a heuristic — it is the time discretization of inertial dynamics on the density manifold.
The structural lesson is about the physics of computation. Attention has been interpreted as a particle system (interacting tokens), a gradient flow (energy minimization), and a kernel method (similarity weighting). Each interpretation suggests different improvements. The particle interpretation suggests SympFormer: if tokens are particles, give them mass. First-order particles are overdamped; second-order particles have inertia. The improvement is not architectural — it is physical. The same interaction energy, the same tokens, the same computational cost. The only change is that the tokens remember where they were going.