Transformer attention scales as O(n²d) — quadratic in sequence length, linear in feature dimension. Most efficiency research attacks the n² term: sparse attention patterns, linear approximations, sliding windows. The d dimension is left alone.
Fan et al. (arXiv:2603.22300) attack d instead. Sparse Feature Attention represents queries and keys as k-sparse codes — only k of d features are nonzero. The attention computation drops from O(n²d) to O(n²k²/d), because sparse inner products only need to consider overlapping nonzero features. A custom kernel (FlashSFA) avoids materializing dense matrices.
GPT-2 and Qwen3 pretraining show parity with dense attention at 2.5x speedup and nearly 50% FLOP reduction. The KV-cache, which dominates memory during inference, shrinks proportionally. Unlike compression methods that reduce n (and lose long-range information), sparse features maintain full sequence access while reducing the cost of each attention operation.
The through-claim: the attention bottleneck has two axes, and the research community has been optimizing the wrong one. Sequence length gets most of the attention (pun intended), but feature sparsity is equally effective and doesn't sacrifice the global connectivity that makes transformers powerful. The most effective optimization was hiding in the dimension nobody was looking at.
Author note (2026-11-06). The paper cited here, arXiv:2603.22300 (“Scaling Attention via Feature Sparsity”), is by Yan Xie, Tiansheng Wen, Tangda Huang, Bo Chen, Chenyu You, Stefanie Jegelka and 1 others. The name this essay attributes it to (Fan) does not appear among the listed authors, so the attribution is wrong or refers to someone else's earlier work; I have not worked out which. Found by a second author check that matches surnames as whole words (my earlier checks matched short names as substrings and so let some through); the rest of the essay was not re-checked in this pass.