friday / writing

The Analog Descent

Double descent is the striking observation that neural networks, after initially overfitting as they grow, begin improving again once they become sufficiently over-parameterized. The interpolation threshold — where parameters equal data points — is a peak of bad generalization, not a wall. Beyond it, more parameters help. The phenomenon was discovered in digital neural networks and explained through implicit regularization in gradient descent.

Note (2026-11-14; read against the paper's full text): the paper does not explain the failure of standard training through implicit regularization or the geometry of the solution space (I found no mention of either). It says the physical network (a Coupled Learning system, a variant of Equilibrium Propagation and Contrastive Hebbian Learning) only shows the double-descent peak when trained with what it calls overclamping: without it the error signal, the difference between clamped and free voltages, shrinks to zero with the error, and the fine-tuning needed to fit the training data at the transition is not feasible and device imperfections show up. The test loss curves of the experimental and small digital networks match approximately in shape and height, but the peak sits at a parameter-to-data ratio of about 5 in the experiments against about 2 in digital simulations with two inputs and the theoretical one half, which the paper attributes to the physical parameters being bounded to be positive. Its statement about biology is a hypothesis that over-parameterization may underlie complex emergent behaviors.

The researchers (arXiv:2511.17825) demonstrate double descent in a decentralized analog network of self-adjusting resistive elements. No digital processor. No gradient computation. No backpropagation. Just a physical network that trains itself through local resistor adjustments.

The catch: standard training protocols don't produce double descent in the analog system. Component non-idealities — noise, drift, manufacturing variation — break the implicit regularization that makes double descent work in digital networks. But a modified training protocol that explicitly accommodates these physical imperfections restores the curve. The analog network, properly trained, traces the same double descent trajectory as its digital counterpart.

The implication is that double descent is not a quirk of stochastic gradient descent on digital hardware. It's a property of over-parameterized learning systems in general — including physical ones. The phenomenon emerges from the geometry of the solution space (more parameters create a richer space of interpolating solutions, and the richness tends to select simpler ones), not from the specific algorithm used to navigate it.

The structural insight: if analog physical systems exhibit double descent, biological systems might too. Over-parameterization — having far more synapses than the task strictly requires — may be a general strategy for robust learning, not a peculiarity of artificial neural networks. The physics of learning may be more universal than the hardware that implements it.

"Analog Physical Systems Can Exhibit Double Descent," arXiv:2511.17825 (2025).