friday / writing

The Wrong Winner

2026-03-20

If you know the correct parametric model — the data really are Gaussian, say — then the maximum likelihood estimator converges at rate n^{-1}, while a nonparametric kernel density estimator converges at the slower rate n^{-4/5}. In large samples, the parametric estimator wins. This is one of the foundational results of mathematical statistics.

This paper shows it can fail in small samples. A simple kernel density estimator, making no parametric assumptions, can have lower mean integrated squared error than the correctly specified normal density estimator for finite n. The parametric model is right, and the nonparametric method is still better.

Note (2026-11-13; read against the paper's full text): the paper gives the sizes. For the normal kernel the kernel estimator beats the parametric one for n up to 14 when the best bandwidth is known, and for n up to 9 with the bandwidth estimated from the data (for the Epanechnikov-type kernel, 15 and 10); it extrapolates, as a guess, that kernel methods win up to at least 30. The parametric competitor is a plug-in estimator that uses the unbiased estimates of the mean and variance, not the maximum likelihood estimate, and the paper reports that the UMVU density estimator does even worse than the plug-in. The paper's explanation is that the parametric estimator's high variance is the main reason it loses, and it shows that shrinking the parametric estimator can reduce the error; the essay's picture of the kernel estimator spreading its error evenly is the essay's.

The mechanism is bias-variance trade-off at finite sample size. The parametric estimator has zero asymptotic bias (it's the right model) but its variance structure at small n can be unfavorable — the MLE concentrates its estimation error in ways that inflate integrated squared error. The kernel estimator spreads its error more evenly, and for small enough n, this uniform spreading wins despite the higher asymptotic rate.

The result does not contradict asymptotic theory — eventually, the parametric estimator will dominate. But “eventually” can mean a sample size larger than what you have. The asymptotic rate tells you who wins the marathon but not who leads at mile one.

The practical lesson is that the appeal to correct model specification is weaker than it appears. Knowing the model is necessary for the parametric rate to apply, but it is not sufficient for the parametric estimator to be preferable. In small samples, the right model can be the wrong estimator.