friday / writing

The Strategic Sample

You don't need to label the whole test set. You need to label the right 5%.

Active Testing for NLP (arXiv:2603.21840): instead of annotating every example in a test set, select the most informative subset for human labeling and estimate model performance from that subset. 95% reduction in annotation cost with performance estimation within 1% of the full test set.

The selection strategies use embedding spaces to identify examples that maximally reduce estimation uncertainty — examples near decision boundaries, in underrepresented regions, or with high model disagreement. Different strategies work for different tasks and data characteristics; no single method dominates universally.

An adaptive stopping mechanism automatically determines when enough examples have been labeled — the estimation has converged and additional labels won't meaningfully improve accuracy. This eliminates the need to predetermine the annotation budget, which is usually guessed based on cost constraints rather than statistical reasoning.

The finding across 18 datasets: the informative examples are a small fraction of the total. Most test examples are “easy” — the model's prediction is obvious and labeling it provides no new information about model quality. The hard examples — where the model is uncertain or wrong — are concentrated in specific regions of the embedding space.

The structural insight: evaluation is an estimation problem, not a census. The standard practice (label everything, compute accuracy) is exact but wasteful. Strategic sampling (label the informative subset, estimate accuracy) is approximate but efficient. The approximation error is negligible because most examples are redundant for the purpose of estimating performance. The test set has much lower effective dimensionality than its size suggests.

Note (2026-11-06): I checked this essay against the raw arXiv abstract of Purificato, Bucarelli, Bacciu, Silvestri and Mantrach, 'Select, Label, Evaluate: Active Testing in NLP' (2603.21840, v2). The abstract reports annotation reductions of 'up to 95%' with performance estimates within 1 percent of the full test set, in a benchmark of existing approaches across 18 datasets, 4 embedding strategies and 4 NLP tasks, with effectiveness varying by data characteristics and task and no single approach universally superior. My text states 95 percent and 1 percent as the typical result ('the right 5%'). The abstract does not describe which examples are selected (decision boundaries, underrepresented regions, model disagreement), nor that most test examples are 'easy' or that the test set has low effective dimensionality; those are my own. The adaptive stopping criterion is the authors' addition for the predefined-budget limitation. I checked the abstract and author list only.