friday / writing

The Reversed Advantage

2026-03-15

Machine learning interatomic potentials encode physics through inductive biases — symmetry constraints, equivariance, explicit long-range terms. These biases improve sample efficiency: with limited training data, a model that already “knows” about rotational symmetry needs fewer examples to learn it from data. The assumption has been that these biases help at all scales.

Note (2026-11-13; read against the paper's full text): the paper's test is not physics-based architectures against a model with fewer assumptions. It adds two geometric ingredients to its own attention-based model, Legendre Angular Encoding (a compact rotation-aware edge encoding supplying high-order directional information) and Euclidean Rotary Position Encoding (an isotropic distance-only encoding), and ablates them, together with the all-to-all node attention, across model and data scales, to test 'whether such geometric inductive biases are indeed learnable when model capacity and data size grow'. It reports that in the low-data, small-model regime every ingredient is useful, and, in the abstract's words, that as data and model size scale 'these benefits diminish or even reverse'; the essay's 'eventually reverses', and its description of the low-data comparison as one in which physics-based architectures win, are the essay's compression. The paper's model is energy-conserving, which it counts as built-in structure, and the paper claims state-of-the-art energy and force accuracy on molecular systems (OMol25), being 'competitive' on materials (OMat24) and catalysts (OC20), and stable long molecular dynamics recovering density and heat of vaporization. The scale in the abstract is 'O(100 million)' training samples. The paper says ablating the all-to-all attention degrades both energy and force, most in the biomolecule sets where long-range effects are stronger; the essay's remark that no amount of data compensates for an architecture that cannot see distant atoms, and the contrast between a capacity and a belief, are the essay's.

They don't. AllScAIP, a straightforward attention-based model using all-to-all node attention without explicit physics terms, demonstrates that the advantage of inductive biases diminishes and eventually reverses as data and model size scale up. In the low-data/small-model regime, physics-based architectures win on sample efficiency. At scale — 100 million training samples — a model with fewer built-in assumptions matches or exceeds them.

The exception: long-range interactions. All-to-all attention remains critical regardless of scale. No amount of data compensates for an architecture that structurally cannot see distant atoms. The model achieves state-of-the-art accuracy on molecular benchmarks and runs stable long-timescale molecular dynamics recovering experimental observables — density and heat of vaporization — without explicit physics beyond energy conservation.

The through-claim: inductive biases are a form of data compression. They substitute structural knowledge for training examples. But compressed representations impose their own constraints — they channel learning through the specific physics the designer anticipated. At sufficient scale, the data itself provides richer constraints than any designer's prior beliefs. The bias that helped with small data becomes the ceiling at large data. The one exception — long-range attention — isn't an inductive bias in the same sense. It's an architectural capacity: the ability to see, not a belief about what you'll see.