Reward models for language model alignment evaluate which of two responses is better. The standard approach: generate a detailed analysis, then extract a verdict. The analysis is expensive — it requires the full reasoning capacity of a large model. For every comparison, the model thinks deeply.
Zhu and colleagues (arXiv:2602.20670) find that the model already knows how confident it is before reasoning. The log-probability margin between the verdict tokens — the gap between the probabilities of “A is better” and “B is better” — correlates strongly with prediction correctness. High margin means the answer is obvious; low margin means it's genuinely uncertain.
CAMEL exploits this. Make a lightweight single-token judgment first. If the confidence is high, stop — the quick answer is almost always correct. If the confidence is low, invoke deeper reflection. The result: a 14B-parameter model outperforms a 70B-parameter model by selectively deploying computation where it's needed. The 3.2-percentage-point accuracy gain comes not from thinking harder everywhere but from thinking harder only where it matters.
The counterfactual training ensures that reflection, when invoked, actually changes the answer rather than confabulating reasons for the initial judgment. The model is trained on prefixes where the initial verdict was wrong, learning to genuinely correct itself rather than rationalize.
The general observation: a system that knows when it's uncertain can allocate computation selectively, outperforming a larger system that treats every case identically. The capacity to recognize difficulty is a resource — it converts a fixed computational budget into a variable one that concentrates effort on hard cases and skips easy ones. The shortcut is knowing when you don't need to try.