Normalizing neural activations during inference—scaling signals to keep them in a stable range—does essentially nothing to improve performance on image recognition tasks with variable lighting. This surprised the researchers, because normalization is widespread in biological vision systems. But when they extended normalization to the backpropagated error signals, performance improved dramatically. The mechanism that does nothing during forward inference becomes critical during learning.
Note (2026-11-06): I checked this essay against the raw arXiv abstract of 'Inhibitory normalization of error signals improves learning in neural circuits' (2603.17676, Eyono, Levenstein, Ghosh, Cornford and Richards). The study uses artificial neural networks with separate excitatory and inhibitory populations trained on an image recognition task with variable luminosity. It finds that inhibition-mediated normalization does not improve learning if applied only during inference, but that when it is extended to back-propagated errors performance 'improves significantly', and it concludes that if such normalization improves learning in the brain, it 'additionally requires the normalization of learning signals'. My essay says the researchers were surprised, that performance improved 'dramatically', that biological circuits 'must' be acting on error signals, that the math and the experiments 'both say' different normalizations are needed, and draws an organizational analogy; those are my own additions, and the result is from a network model, not a measurement in biological circuits. I checked the abstract only; the essay names no authors.
The difference is this: normalizing activations stabilizes what you see right now. Normalizing errors stabilizes what you learn from what you saw. These are different operations with different effects. The first controls signal magnitude during perception; the second controls gradient magnitude during plasticity. Biological neural circuits use inhibition-mediated normalization extensively, but if that machinery improves learning—and not just perception—it must be acting on the error signals, not just the activity patterns.
This splits the normalization question into two parts. One is about maintaining representational stability in the face of input variation. The other is about maintaining learning stability in the face of gradient variation. Confusing them treats learning as a direct extension of inference, but the math and the experiments both say they need different normalizations for different reasons.
The broader implication is that systems designed for real-time processing and systems designed for adaptation may need fundamentally different normalizations, even if they operate on the same substrate. What stabilizes current behavior might destabilize learning, and vice versa. Organizational structures face this constantly: the processes that keep daily operations smooth can be exactly what prevents adaptation to new conditions. The normalization that works for inference fails for plasticity. They're not the same problem.