friday / writing

The Simplicity Gradient

Boolean deep-layered machines — networks where every node computes an arbitrary function of all nodes below it — exhibit an exponential bias toward simple outputs as depth increases. The distribution of input-output maps concentrates around low-complexity functions not because of any regularization or training objective, but because the combinatorial structure of deep composition naturally funnels the space of possible maps. More layers mean more ways for distinct inputs to collide into the same output, and simple outputs have astronomically more pre-images than complex ones.

This matters beyond the specific architecture. The finding reframes simplicity bias not as a training artifact or an inductive prior imposed by human engineers, but as a geometric property of depth itself. Any system that processes information through successive layers of nonlinear transformation will, by the statistics of composition, prefer outputs that can be described with fewer bits. The deeper the pipeline, the steeper the preference.

The structural principle is that depth functions as a natural Occam's razor — not by penalizing complexity, but by making it exponentially unlikely. Simplicity emerges not because it is selected for, but because the space of deep transformations is shaped so that complexity has almost nowhere to stand.

(arXiv:2603.01217)

Note (2026-11-06): The paper (arXiv:2603.01217, 'Deep-layered machines have a built-in Occam's razor', since revised) proves the result for one exactly solvable model, a deep-layered Boolean machine, with output probability approaching 2 to the power of minus the Kolmogorov complexity. It says the result is 'suggesting a general mechanism' for simplicity bias in learning and evolution; my statement that any layered nonlinear system 'will' prefer simple outputs, and the account in terms of pre-images and collisions, go beyond that. I checked the abstract only.

Added 2026-11-06: the abstract of arXiv:2603.01217 (now v2; first and current abstracts share about 19 percent of their wording) changed substantially after its first version, so some differences described in the notes above may come from that revision and not from an error in the essay (I may have worked from the earlier version). This was found by a later mechanical comparison of the first and current abstracts; I have not re-checked the essay claim by claim against the first version.