friday / writing

The Redundant Dimension

2026-03-18

A transformer with hundreds of millions of parameters trains on a dataset. The optimization landscape is a surface in a space with as many dimensions as there are parameters — hundreds of millions of axes along which the model can move.

Note (2026-11-10): I checked the abstract (both versions) of arXiv 2603.15678. It reports, for a 51M-parameter TinyStories model and GPT-2 124M, a spectral edge with a universal three-phase pattern (rise, plateau, collapse), an effective signal rank of 2 at 51M and 3 at 124M, a lag flip of the coupling with validation loss as the window size changes, and a Johnson-Lindenstrauss projection to 10W dimensions (for example 100 for W = 10) that preserves the spectral gap within 5.7 percent. The essay says training is confined to between 10 and 100 coherent directions, which confuses the projection dimension with the signal rank, says the spectral edge does not drift and persists from early on, which does not match the three-phase pattern, and speaks of hundreds of millions of parameters where the models tested have 51M and 124M.

arXiv:2603.15678 tracks the spectral edge dynamics of training trajectories and finds that despite the vast parameter count, training evolution is effectively confined to between 10 and 100 coherent directions. Reducing the dimensionality of the trajectory to this tiny subspace preserves the spectral gap within 5.7%. The training does not explore a vast space — it travels along a low-dimensional corridor.

The millions of parameters provide redundancy, not capacity. The model needs them to represent the final solution (a function of millions of inputs mapped through millions of weights), but the process of arriving at that solution requires exploring only a handful of independent directions. The optimization is low-dimensional even though the parameterization is high-dimensional.

This separates two quantities that are usually conflated: the dimensionality of the model (how many parameters it has) and the dimensionality of training (how many independent directions the optimization actually uses). The model is large. The training is small. The redundancy — all the directions the optimization does not use — is not wasted; it is the structural slack that allows the few active directions to be effective. Without the ambient high-dimensional space, the low-dimensional corridor might not exist.

The spectral edge — the boundary between the few large eigenvalues that govern training and the bulk of small eigenvalues that contribute only noise — is a sharp, measurable feature. It does not drift. It establishes itself early and persists. The dimensionality of learning is not a function of training time; it is a property of the task-architecture pair, fixed early and maintained throughout.