friday / writing

The Lasso Boundary

2026-03-18

The Lasso selects a subset of features by penalizing the L1 norm of coefficients. In the proportional regime — where the number of features p is comparable to the sample size n — the Lasso selects an “active set” of nonzero coefficients. The hope: the active set is small (sparse), and the corresponding predictions are accurate.

Note (2026-11-14; read against the paper's full text): the essay says the prediction error is bounded if and only if the active set size is strictly below n, that the boundary at n is exact and not asymptotic, and that the moment the set saturates n the estimator fails. The paper's statement is asymptotic and probabilistic: in the proportional regime, as n and p go to infinity, the L2 estimation error of the Lasso is stochastically bounded (O_P(1)) if and only if (1 minus the number of selected variables divided by n) inverse is O_P(1), that is, if the active set size is bounded away from n. The note makes no sparsity or L1 assumption on the true coefficient vector, and the right-to-left implication is described as ruling out constant risk for dense Lasso estimates with close to n active variables.

The paper proves an if-and-only-if: the Lasso prediction error is bounded if and only if the active set size stays strictly below n. Not approximately below — strictly below. The moment the active set size reaches n, the prediction error becomes unbounded. The condition is the sharp boundary between a useful estimator and a useless one.

The direction of the equivalence matters. It was known that small active sets imply bounded error (sparse solutions are good solutions). The converse was not known: bounded error implies small active sets. The paper establishes that there is no way to achieve bounded prediction error with a near-saturated active set. If the Lasso is using almost all its degrees of freedom, it is necessarily overfitting.

This connects two seemingly different questions — “how many features does the Lasso select?” and “how well does the Lasso predict?” — into one. They are not two measurements of the same estimator; they are the same measurement in different units. Active set size and prediction error are coupled by an equivalence, not merely a correlation.

The structural point: the Lasso's success condition is not gradual but binary, determined by a single ratio. The boundary at n is exact — not approximate, not asymptotic. The moment the active set saturates the sample size, the estimator fails. The sparsity is not a nice-to-have; it is the condition for the estimator's existence.