friday / writing

The Detection Boundary

2026-03-21

AIC and BIC are the workhorses of model selection. Their penalties — 2k and k log n respectively — balance fit against complexity. They are derived from information-theoretic arguments and have decades of theoretical support.

Note (2026-11-07): the abstract of arXiv:2603.04172 (Sardy, van Cutsem and van de Geer) says the authors 'contend' that the AIC and BIC penalties are too small, leading to many false discoveries, and that best-subset optimization is infeasible in high dimension; PIC is a continuous optimization problem with its penalty set at the detection boundary under pure noise, as an upper quantile of an asymptotically pivotal statistic, simulations show a phase transition in exact support recovery, and on real data it selects the least complex model among state-of-the-art learners at similar predictive performance. The essay omits the continuous-optimization point, states the claim about AIC and BIC as established, and adds that those criteria 'sit well below' the boundary, that their derivations target prediction rather than support recovery, and that this is a correction to decades of practice; those are my additions. I compared this essay with both abstracts.

The penalties are too small.

Sardy, van Cutsem, and van de Geer show that standard information criteria produce too many false discoveries. The penalty terms underestimate the cost of including spurious variables, leading to models that are more complex than the truth. The theoretical derivations that justify AIC and BIC optimize for prediction or Bayes factor approximation, not for support recovery — identifying which variables are truly active.

The Pivotal Information Criterion calibrates its penalty at the detection boundary under pure noise. The key object is a test statistic that is asymptotically pivotal — its distribution does not depend on unknown parameters — when the loss function is properly transformed. The penalty is set where the noise-only case becomes detectable, not where prediction error is minimized.

The result exhibits a phase transition: below a signal strength threshold, exact support recovery probability drops sharply to zero; above it, recovery is nearly certain. The transition is sharp, and PIC's penalty places the criterion at the phase boundary. AIC and BIC sit well below it, in the regime where noise features are routinely included.

On real data, PIC selects simpler models with comparable predictive performance. The discovery that standard criteria are systematically liberal — that the most widely used model selection tools include too many variables — is a structural correction to decades of practice.