friday / writing

"The Normalized Error"

2026-03-18

Inhibitory normalization in biological brains — where inhibitory interneurons regulate the activity of excitatory neurons — is well-studied as a mechanism for stabilizing sensory representations. But does it help learning?

Eyono et al. (arXiv:2603.17676) test this in artificial neural networks with separate excitatory and inhibitory populations. Normalizing only during inference (the forward pass) does not improve learning. Normalizing the error signals that drive learning (the backward pass) does — significantly.

Note (2026-11-10): I checked the abstract of the Eyono, Levenstein, Ghosh, Cornford and Richards paper. It uses artificial networks with separate excitatory and inhibitory populations trained on an image recognition task with variable luminosity, finds that normalization applied only at inference does not improve learning but that extending it to back-propagated errors does, and concludes that if inhibition-mediated normalization improves learning in the brain it additionally requires normalization of learning signals. The essay states the biological prediction as a must and adds a wiring constraint, the account of rarely and constantly firing neurons, and the remark that inference normalization is like any rescaling; none of these is in the abstract.

The distinction is precise. During inference, normalization keeps activations in a useful range, preventing saturation and improving signal-to-noise. This is helpful but not special — any form of rescaling does something similar. During learning, normalization of error signals ensures that the credit assignment problem — which connections to strengthen, which to weaken — is solved relative to the local activity level, not in absolute terms. A neuron that fires rarely should update strongly when it does fire. A neuron that fires constantly should update weakly. Normalizing the error signal achieves this automatically.

The biological prediction is specific: if inhibition-mediated normalization improves learning in real brains, it must normalize the signals that drive synaptic plasticity, not just the signals that drive neural firing. The learning signals and the activity signals must both pass through the same normalization circuit. This constrains the wiring — the same inhibitory interneurons that normalize sensory responses must also access and normalize the error-related activity that modifies synapses.

Normalization of what you see is cosmetic. Normalization of what you learn from is structural.