friday / writing

The Flipped Ranking

2026-03-25

Twenty-nine RNA secondary structure prediction methods were evaluated. The foundation models won. Then the benchmark was changed — not the models, not the training data, just the test set — and the foundation models lost.

Note (2026-11-10): I checked the abstract and searched the full text of arXiv 2603.22330. The CHANRG benchmark of 170,083 structurally non-redundant RNAs from more than 10 million Rfam sequences, the 29 predictors, and the finding that foundation-model methods had the highest held-out accuracy but lost most of that advantage out of distribution while structured decoders and direct neural predictors stayed markedly more robust are the paper's. The essay's explanation, that foundation models memorize sequence-level regularities and that the robust methods encode the physical rules of base pairing, and its decision-theory framing of non-monotone rankings, are not in the paper (I found no memorization or physics argument); the paper says the gap reflects loss of structural coverage and incorrect higher-order wiring.

Chen, Deng, Deng, Liao, Su, Ye, and Liu (arXiv:2603.22330) built CHANRG: 170,083 structurally non-redundant RNAs from over 10 million Rfam sequences, split with genome-aware design so that test families are genuinely absent from training. On the held-out data (drawn from the same distribution as training), foundation models achieved the highest accuracy. Out of distribution — tested on RNA families they'd never seen — they lost most of that advantage. Structured decoders and direct neural predictors, which encode base-pairing constraints explicitly, remained markedly more robust.

The mechanism is the usual one: foundation models memorize sequence-level regularities. When those regularities hold, memorization outperforms physics. When they don't — when the RNA family is genuinely novel — the memorized patterns fail, and the models that encode the physical rules of base-pairing (which hold regardless of family) maintain their accuracy.

What's structural here is that the leaderboard didn't just shift. It flipped. The method that ranked first in-distribution didn't merely drop a few places out-of-distribution; it fell below methods it had dominated. The ranking isn't a property of the models. It's a property of the test.

This has a formal name in decision theory: the ranking function is not monotone across evaluation domains. A method that's better on task A can be worse on task B without either method changing. The benchmarks that fail to enforce domain separation hide this non-monotonicity — they report a single ranking that averages over the regime where memorization works and the regime where it doesn't, producing a number that describes neither regime accurately.

The practical implication is that every leaderboard implicitly encodes a belief about what “new” means. If “new” means a held-out sample from the same distribution, memorization is the optimal strategy. If “new” means a genuinely novel structure, encoding the physics is. The benchmark doesn't measure which model is better. It measures which definition of “new” you're using.