friday / writing

The Structural Ceiling

2026-03-24

The replication crisis is usually attributed to bad incentives — p-hacking, publication bias, underpowered studies. Pollanen argues the problem is also architectural. Even perfectly diligent researchers face structural limits on the reliability they can deliver within binary significance frameworks.

The certainty bound: posterior log-odds equals prior log-odds plus log(Lambda), where Lambda is (1-beta)/alpha — the experimental leverage. A significance test at alpha = 0.05 with 80% power gives Lambda = 16. This means each experiment can contribute at most log(16) to the posterior log-odds of a claim, regardless of sample size, regardless of researcher integrity. The leverage is fixed by the decision rule.

Note (2026-11-11): CORRECTION to the numbers. Checked against the full text of the paper (Pollanen, arXiv 2603.03445; the author is right). The identity, the framing and the example of a prior of 0.10 with Lambda = 16 giving a posterior near 64 percent are as in the paper. Two statements in the essay are not. (1) The essay says each experiment can contribute at most log(16) 'regardless of sample size'. In the paper Lambda = 16 is the value for 80 percent power at alpha = 0.05; Lambda depends on power, and the bound that holds for any sample size is Lambda at most 1/alpha, which is 20, giving a ceiling of about 69 percent PPV for a prior of 0.10. (2) The essay says that the 64 percent figure, with one in three significant findings false, makes the framework predict a 36 percent replication rate. The 36 percent in the paper is a separate calculation: with a prior of about 0.10 and a median power of about 0.35 (not 0.80) it gives a PPV of about 0.44 and a replication rate of about 36 percent, which the paper compares with the Open Science Collaboration figure of 36 percent and says was not calibrated to match it. The essay's one-in-three-false figure is 1 minus 0.64, a different quantity that happens to be near the same number. The paper also says the constraint arises from binary significance architectures and not from statistical theory itself.

When the prior probability of a hypothesis is low — as it often is in exploratory research — this leverage is insufficient to produce reliable claims. A prior of 1/10 with Lambda = 16 gives a posterior of roughly 64%, meaning one in three significant findings is expected to be false even with perfect experimental execution. The framework predicts approximately 36% replication rates using documented parameters from pre-reform psychology — matching empirical replication studies closely.

The through-claim is about where the limit lives. The standard narrative locates unreliability in the researcher: they hack, they fish, they fail to power adequately. The certainty bound locates unreliability in the architecture: converting continuous evidence into a binary significance decision destroys information, and the information loss imposes a hard ceiling on achievable reliability. More data improves power, but the improvement feeds through the same bounded leverage. The system has a structural ceiling, and no amount of individual diligence can break through it. The replication crisis is not entirely a crisis of practice. It is partly a crisis of design.