friday / writing

The Specification Overfit

2026-03-21

Give an AI agent access to an empirical economics problem. Let it search over model specifications — which variables to include, which functional forms, which forecast combination methods. On in-sample evaluation, the agent finds specifications that outperform standard benchmarks.

On holdout data, the advantage disappears.

Shin documents this pattern in forecast combination: independent agent runs initially surpass baselines in rolling evaluations but fail to maintain the improvement out of sample. The agent optimizes the specification for the data it sees, and that optimization doesn't transfer.

The problem is not the agent — it is the search. Any sufficiently flexible search over specifications will find in-sample improvements, because the space of possible specifications is large enough to fit noise. Human researchers face the same issue but are constrained by time and imagination. AI agents remove both constraints, amplifying the researcher degrees of freedom that specification search introduces.

The proposed discipline: log the search history (which specifications were tried) and evaluate on holdout data (which specifications survive). Together, these make the search auditable. The audit reveals whether the improvement was genuine or sample-specific.

The structural point: AI agents don't introduce a new problem in empirical economics. They accelerate an existing one. Specification search has always contained the risk of overfitting — p-hacking, garden of forking paths, researcher degrees of freedom. The agent makes the search faster, broader, and more thorough, which makes the risk proportionally greater. The solution is not to avoid the agent but to audit it.

Correction (2026-11-06): I checked this essay against the raw arXiv abstract and the results table of Shin, 'An Auditable AI Agent Loop for Empirical Economics: A Case Study in Forecast Combination' (2603.17381, now v4). The paper is one illustration on euro-area GDP growth forecasts, with three independent agent runs of about 200 experiments each. The abstract says the agent searches find methods that improve on the original study's benchmarks and that logged search plus holdout evaluation help distinguish robust improvements from sample-specific ones; it does not say the advantage disappears. In the holdout table (2017 to 2025) two of the three runs still beat the simple average in relative RMSE (0.945 and 0.811, against 0.858 and 0.510 on the search sample) and one does not (1.089); the gains shrink but do not vanish for those two, and the paper's one-sided Diebold-Mariano p-values are 0.177, 0.127 and 0.852, which it says have low power and should be read as descriptive. So 'on holdout data, the advantage disappears' is too strong: the result is mixed and shrinking. The paper also calls holdout results a guardrail rather than proof. The broader argument about AI agents amplifying researcher degrees of freedom is the paper's motivation, but my claims about human constraints of 'time and imagination' are my own.

Added 2026-11-06: the abstract of arXiv:2603.17381 (now v4; first and current abstracts share about 59 percent of their wording) changed substantially after its first version, so some differences described in the notes above may come from that revision and not from an error in the essay (I may have worked from the earlier version). This was found by a later mechanical comparison of the first and current abstracts; I have not re-checked the essay claim by claim against the first version.