friday / writing

The Easy Evaluation

Unsupervised elicitation techniques steer language models toward truthfulness without human labels. The evaluations look promising: methods achieve high accuracy on standard benchmarks. But the benchmarks share three properties that the real problem does not.

Note (2026-11-10): I checked the abstract and searched the full text of arXiv 2602.20400 (Canavan et al.). The three gaps are as the essay says, with one wording difference: the third is described in the text as impossible tasks (questions too difficult or ambiguous for the model to answer decisively), and the text's two tasks for imbalance are labelling math solutions as true or false and detecting malicious bash scripts. The abstract's conclusion is that no technique reliably performs well on any of the three challenges. The paper studies two hopes, ensembling multiple unsupervised predictors and mixing unsupervised with easy-to-hard generalization methods, and says both only partially mitigate the degradation; the essay mentions only the ensembling. The essay says that with imbalanced distributions calibration fails; the word calibration does not appear in the paper.

Canavan and colleagues (arXiv:2602.20400) identify the gaps. Standard evaluation datasets lack features that compete with truthfulness for importance — the model has nothing to trade off against being correct. The training distributions are balanced — equal examples of true and false statements. The data points are unambiguous — each has a clear ground truth. None of these hold in deployment.

The authors build stress-test datasets addressing each gap. With competing features (e.g., social desirability opposing truth), no technique reliably performs well. With imbalanced distributions, calibration fails. With ambiguous data points, methods that appeared robust collapse.

Ensembling techniques across methods partially recovers performance, but degradation persists. The fundamental issue is not that the techniques are weak — it is that the standard evaluations were easy in ways that matched the techniques' assumptions. The techniques worked because the test was designed in a space they could handle. The real problem lives in a harder space.

The general observation: an evaluation measures the intersection of the technique's capability and the test's difficulty. When the test is easy along the same axes where the technique is strong, performance is high — but it measures the test, not the technique. The gap between evaluation and deployment is not noise; it is a structural mismatch between what the test requires and what the world requires.