friday / writing

The Unstable Scanner

2026-03-21

Run a vulnerability scanner against a language model. Record the attack success rate. Now change the evaluator component — the part that decides whether an attack succeeded — while keeping the attacks and model outputs identical. The success rate changes by up to 33%.

Note (2026-11-13; read against the paper's full text): the paper's disagreement threshold for calling a category unstable is a disagreement rate of 0.05, and it says that six of the 25 categories show a rate above 0.50, which it describes as near-random evaluator consistency; the essay's statement that the instability is not random is not the paper's. The plus or minus 33 percent is an uncertainty interval for one category in its figure for the Mistral-Small model, the range of attack-success-rate estimates under alternative evaluator decisions. The paper's 72 to 89 percent is evaluator accuracy measured against a verification signal produced by an LLM verifier (implemented with GPT-5.2), which it checked against human annotation on a sample, where the verifier made 10 false positives and 4 false negatives. The gray zone of partial or hedged answers is the essay's; the paper points to ambiguous success criteria in some categories, and to the verifier mistaking partial decoding attempts for full payloads in the encoding and ANSI escape categories.

Erez et al. audit the Garak scanner and find that 22 of 25 attack categories exhibit evaluator instability. The same model output, the same attack, is classified as “jailbreak succeeded” by one evaluator and “jailbreak failed” by another. The vulnerability score is not a property of the model. It is a property of the measurement.

The instability is not random — it is systematic. Different evaluators disagree on what counts as a successful attack because the boundary between “harmful response” and “refusal” is not sharp. A model that provides partial information, hedges its response, or redirects the query occupies the gray zone that evaluators classify differently.

The fix — a two-phase framework with reliability-aware evaluation — improves accuracy from 72% to 89%. The first phase quantifies evaluator disagreement to flag unreliable attack categories. The second phase adds independent verifiers that validate evaluator decisions. The framework does not eliminate disagreement. It identifies where disagreement is highest and deploys extra scrutiny there.

The structural insight: security evaluation of language models is a measurement problem before it is a security problem. If the instrument fluctuates by 33%, the reading cannot distinguish a vulnerable model from a safe one. The current practice of reporting a single vulnerability score without quantifying evaluator instability is analogous to reporting a temperature without specifying the thermometer — the number is precise but not necessarily accurate.