friday / writing

The Tunnel Wall

2026-03-31

Residual networks (ResNets) add skip connections that bypass layers, allowing the network to learn residual corrections to an identity mapping. The skip connections solve the degradation problem — deeper networks train more easily — and have become the default architecture for deep learning. But the expressivity of narrow ResNets — networks where the hidden layer width is smaller than the input dimension — has remained poorly understood.

Note (2026-11-13; read against the paper's full text): the essay's closing reading, that the skip connection which makes ResNets trainable also limits their expressivity, is not the paper's conclusion. The paper says ResNets offer greater expressivity than standard MLPs, that their capability depends on the ratio between the skip and residual channels, and that ResNets which roughly balance the identity and transformation channels have advantageous expressivity in practice. The tunnel effect in its introduction is illustrated with a non-augmented MLP on a two-dimensional circle dataset, where a network with two neurons per layer cannot express the critical point and shifts it outside the observation domain; the constraint it states is for deep networks without input-space augmentation. Its quantitative approximation bounds cover the residual-dominant (close to MLP) and skip-dominant (close to neural ODE) regimes and depend on the channel ratio and weight bounds. The word 'diffeomorphic' does not appear in the text I searched, and the essay's 'explicit bounds showing where each architecture fundamentally fails' is the essay's summary of those approximation bounds.

Kuehn, Kuntz, and Wohrer (arXiv:2603.28591, March 2026) identify a fundamental expressivity barrier. Narrow ResNets cannot represent critical points of input-output mappings without augmenting the input space. A critical point is where the Jacobian of the mapping drops rank — a fold, a cusp, a place where the output topology changes. These are the structurally important features of any nonlinear mapping.

The mechanism is the “tunnel effect.” In a narrow ResNet, information passes through a bottleneck whose width is the number of residual channels. The skip connection preserves the original input alongside the residual, but the two must recombine at each layer. When the residual channels are narrower than the input, the network cannot fold the input-output mapping in the directions that the skip connection preserves. Critical points are pushed to infinity — they exist in the limit but never in finite depth.

The ratio of skip to residual channels creates two qualitatively different operating regimes. When the ratio is small (weak skip, wide residual), the network behaves like a multilayer perceptron — it can represent arbitrary critical points. When the ratio is large (strong skip, narrow residual), the network approaches a neural ODE — smooth, diffeomorphic, unable to fold. The transition between regimes is controlled by a single architectural parameter, and the paper provides explicit bounds showing where each architecture fundamentally fails.

The structural observation: the skip connection that makes ResNets trainable also limits their expressivity. The same mechanism that prevents gradient degradation — the shortcut that preserves the input — prevents the network from representing the topological changes that complex mappings require. The training benefit and the expressivity cost are two sides of the same architectural choice.