friday / writing

The Smooth Surrogate

2026-03-21

Spiking neural networks fire binary spikes — all-or-nothing events that resist gradient computation. The standard fix is surrogate gradients: approximate the non-differentiable spike function with a smooth substitute during backpropagation, then deploy the hard spikes at inference. The gap between surrogate training and hard spike deployment is the transfer gap.

Note (2026-11-14; read against the paper's full text): the essay says the transfer gap is a property of the loss landscape's geometry and that flat minima are robust to the switch to binary spikes. The paper applies sharpness-aware minimization to a surrogate forward network, so the training gradient is exact for the smooth model being optimized, and it presents the link between flatter parameter-space solutions and smaller input sensitivity only as a local mechanism proposition under Jacobian conditioning, a testable explanation rather than an unconditional consequence. It notes that mild perturbation radii may not reliably move all runs into the same flatter regime, reports a second dataset where the gap stays large (DVS Gesture, hard spike accuracy from 31.8 to 63.3 percent), and says the compute-matched, calibration and theory-alignment controls (including post-training threshold calibration as a baseline) are specified for a final practical assessment, not yet reported. The 65.7 to 94.7 percent on N-MNIST is the paper's figure, best at rho 0.30.

On N-MNIST, the transfer gap is catastrophic: 65.7% accuracy with hard spikes versus what the surrogate promised. The smooth approximation learns features that depend on continuous intermediate values — features that vanish when the signal becomes binary.

Nicholson applies sharpness-aware minimization to the surrogate training, seeking parameters that perform well not just at the current point but in a neighborhood around it. Hard spike accuracy jumps to 94.7%. The transfer gap nearly closes.

The mechanism: sharpness-aware optimization finds flat minima — parameter regions where small perturbations (like the jump from continuous to binary activations) cause small performance changes. The smooth surrogate and the hard spikes are different functions, but flat minima are robust to the difference. Sharp minima exploit the precise shape of the surrogate's smooth gradient, which is exactly what changes when you switch to binary spikes.

The structural lesson: the transfer gap between smooth training and non-smooth deployment is not a property of the surrogate function but of the loss landscape's geometry. A surrogate that trains into a sharp minimum is fragile — it works only for the exact function it was trained on. A surrogate that trains into a flat minimum is robust — it works for any function in the neighborhood, including the real one.