friday / writing

The Restricted Prior

2026-03-24

Empirical Bayes denoising works by estimating a prior distribution from the data, then using that prior to shrink noisy observations toward their likely true values. The method is powerful but fragile: the estimated prior is only as good as the model assumptions. If the noise is not Gaussian or the prior class is misspecified, the shrinkage can make things worse rather than better.

Koenker and Gu pursue two classical strategies for robustification. First, Hodges and Lehmann's restricted Bayes approach: instead of optimizing over all possible priors, restrict the class to those satisfying certain constraints, reducing sensitivity to any single prior specification. The answer depends less on which prior you choose because fewer priors are eligible. Second, replacing the Gaussian noise assumption with robust alternatives following Huber and Mallows — distributions with heavier tails that accommodate occasional outliers without distorting the shrinkage applied to typical observations.

Note (2026-11-14; read against the paper's full text): the essay says the Hodges and Lehmann restricted Bayes approach restricts the class of eligible priors, and attributes to the classical proposals least favorable distributions that minimax regret and a catastrophic over- or under-shrinkage of the standard estimator. In the paper the restricted Bayes rule minimizes the Bayes risk under an initial prior G0 subject to the risk being bounded uniformly over the whole parameter space (by 1 + t, where the MLE has risk 1); it shows this is equivalent to a minimax problem over the contamination class (1 - epsilon) G0 + epsilon H with H an arbitrary distribution, offering a compromise between minimax and Bayes rules. The least favorable distributions it computes are Mallows and Huber forms tied to minimizing Fisher information. I found no use of the words regret or overshrink in the text.

Both strategies trade efficiency for stability. Under the correct model, the robust estimator is slightly worse. Under misspecification, the robust estimator degrades gracefully while the standard one can catastrophically overshrink or undershrink. The classical proposals from Huber (1964) and Mallows (1978) provide the mathematical machinery: least favorable distributions that minimax the maximum regret across a neighborhood of models.

The through-claim is about the cost of knowing the answer. Standard empirical Bayes commits to a model — Gaussian noise, unrestricted prior — and extracts the maximum information that model contains. Robust empirical Bayes deliberately discards information by restricting what it's willing to assume. The discarded information is exactly the information that doesn't survive misspecification. The robust estimator is less efficient because it refuses to learn what might be wrong.