When a neural network perfectly fits noisy training data, it has memorized the noise. By classical learning theory, this should catastrophically degrade test performance. The model has learned the noise as if it were signal, and its predictions on new data should reflect this confusion.
Double descent shows this does not happen. Test performance initially degrades as the model begins to overfit, then recovers and improves as the model capacity grows further or training continues. The network memorizes the noise and generalizes well, simultaneously.
Note (2026-11-15; read against the paper's full text): the essay's waste channel, a single large activation that absorbs the noise, is not what the paper reports. The paper finds two separate things: the internal activations of noisy and clean training data become increasingly separated in the outer layers as training progresses (which it says enables the model to overfit only the noisy data), and, separately, a single very large activation in the shallow layers (found only up to the third layer) that evolves with re-generalization and whose magnitude 'correlated with input patterns but not with output patterns'. I found no statement that the large activation absorbs or carries the noise; the paper links the three phenomena (double descent, benign overfitting, large activation) as empirical findings. The experiments are fully connected networks of three sizes on CIFAR-10 with 30 percent label noise, which the essay does not mention.
The mechanism is not tolerance. The network does not “put up with” noise or average it out. It actively segregates signal from noise using extreme activations.
A single very large activation emerges in the shallow layer across all tested models during the double descent regime. This activation grows as training progresses through the overfitting phase, and its growth correlates with improved generalization. The activation is not a pathology — it is a structural feature. The network creates an internal waste channel: a representational pathway that absorbs the noise component of the data, leaving the remaining capacity available for clean signal representation.
The decomposition is functional. The noise flows into the extreme activation channel; the signal flows through the remaining activations. The two streams are separated not by explicit architecture but by the training dynamics themselves. Gradient descent, continuing past the overfitting threshold, discovers that the cheapest way to fit noise while maintaining generalization is to partition its representational capacity — dedicating one direction to waste and using the rest for signal.
This reframes what “overfitting noise” means. The network is not confused about which data points are noisy. It has identified the noisy component and routed it into a disposal system. The memorization of noise and the generalization of signal are not contradictory objectives but complementary parts of a learned decomposition. The model that fits everything correctly has not failed to distinguish signal from noise. It has distinguished them and filed them separately.
Overfitting is not the failure mode. It is the mechanism by which the network builds its filter.