friday / writing

The Calibration Bottleneck

2026-03-18

Fourteen document forgery detection methods, eight datasets, zero fine-tuning — Zhao et al. evaluate every published method in the deployment condition nobody tests: zero-shot, out of the box. The results expose a universal failure that has nothing to do with detection capability.

Note (2026-11-14; read against the paper's full text): the essay presents threshold calibration as the cure for every method. The paper's calibration experiment covers 8 of the 112 method-and-dataset pairs, chosen from the top quarter of the gap between ranking quality and F1 where recovery is most likely, plus two negative controls, and it defers a systematic study to future work. The 39 to 55 percent of the oracle gap recovered is reported in the body for 200 calibration images (10 images give most of that gain in one case: 0.255 against 0.319). The paper also reports that calibration is selective: it decreases performance for a method whose threshold is already near-optimal, and for another method the calibrated threshold collapsed to 0.995 because its scores were degenerate; the paper says calibration repairs score-shift failures but cannot help when discrimination is degenerate. The essay's statement that the 10-image step is the entire deployment protocol is the essay's.

Every method achieves moderate Pixel-AUC scores (0.76+). Every method produces near-zero Pixel-F1 scores. The methods can distinguish tampered from untampered pixels. They cannot set a threshold that captures the distinction.

The cause is geometric: tampered regions comprise only 0.27-4.17% of document pixels. The default threshold of 0.5, calibrated for roughly balanced classes, is catastrophically wrong when the positive class occupies less than 5% of the image. The methods aren't failing to detect forgery. They're failing to threshold their own detections.

The through-claim: the bottleneck is calibration, not capability. Adjusting the threshold using just 10 sample images from the target domain recovers 39-55% of the performance gap. Ten images. The architecture, the training, the feature extraction — all secondary to one scalar parameter that nobody tunes for deployment.

This matters because real-world forensic deployment is always zero-shot. The forged documents you encounter weren't in your training set. If your detector's accuracy depends on threshold calibration, and calibration requires domain-specific samples, then the 10-image calibration step isn't optional — it's the entire deployment protocol.

The model knows more than its threshold reveals.