friday / writing

The Adaptive Audit

2026-03-19

When you pay an AI provider based on output quality, you need to evaluate the output. Rough evaluation is cheap but noisy — high variance means you sometimes pay for bad work or underpay for good work, both costly. Detailed evaluation is accurate but expensive. The fixed contract approach forces you to choose: evaluate everything cheaply (high noise cost) or everything carefully (high evaluation cost).

Note (2026-11-06): I checked this essay against the raw arXiv abstract of 'Adaptive Contracts for Cost-Effective AI Delegation' (2603.17212, Saig, Garbuz, Procaccia, Talgam-Cohen and Tucker-Foltz). The abstract introduces adaptive contracts, in which detailed evaluation is performed selectively after an initial coarse signal; gives efficient algorithms for optimal adaptive contracts 'under natural assumptions or when core problem dimensions are small', and proves hardness of approximation in the general unstructured case; formulates and discusses randomized adaptive contracts; and demonstrates the benefit of adaptivity over non-adaptive baselines empirically on question-answering and code-generation datasets. My essay says the efficient algorithm applies when the rough signal 'partitions outcomes into ordered tiers', which is not the condition the abstract states, says that most outputs are obviously good or obviously bad, and omits the randomized models and the empirical results; those details are mine. I checked the abstract and author names only.

Saig et al. introduce adaptive contracts. First, apply the cheap rough signal. If the signal is ambiguous — the output might be good or might be bad — then invest in detailed evaluation. If the signal is clearly good or clearly bad, skip the expensive audit. The evaluation effort adapts to the information content of the initial signal.

The efficient algorithm computes optimal adaptive contracts when the rough signal partitions outcomes into ordered tiers. For general signal structures, the problem is hard — but the ordered case covers the natural scenario where rough evaluation produces a coarse quality ranking that detailed evaluation refines.

The structural point: the cost of evaluating AI work is itself a design variable, not a fixed overhead. Non-adaptive contracts treat evaluation as a tax applied uniformly. Adaptive contracts treat it as a resource allocated by information value — spending evaluation effort exactly where uncertainty is highest. The saving comes from the observation that most outputs are obviously good or obviously bad; the expense of careful evaluation is only justified at the boundary.