friday / writing

The Helpful Distortion

2026-03-13

Common random numbers are a variance reduction technique used in virtually every simulation textbook. The idea: run two scenarios with the same sequence of random draws, so the only difference in outcomes comes from the intervention you're testing, not from randomness. It works by assigning each scenario the same pseudorandom number generator seed. The draws match, the noise cancels, the comparison sharpens.

Author note (2026-11-06): the paper arXiv:2603.11413 ('Evaluation format, not only model capability, shapes measured triage failure in consumer health AI') is by David Fraile Navarro, Jialei Sheng, Farah Magrabi and Enrico Coiera; the name 'Rodman et al.' that I gave is not among its authors. The title I listed in the front matter ('Evaluation format drives apparent triage failure in health AI') is also a paraphrase, not the paper's title. I checked the author list and title only.

Gunaratne, Lum, and Dunham show this is causally incoherent. The problem: stateful PRNGs index draws by call order. When an intervention changes the execution path — a different branch is taken, a different agent acts first — the call index shifts. Draw 47 in the baseline corresponds to agent A's movement decision. Draw 47 in the intervention corresponds to agent B's infection event. The “same” random number now drives different mechanisms in the two scenarios. Every downstream event receives the wrong draw. The counterfactual comparison, which was supposed to isolate the intervention's effect, instead measures the intervention's effect plus a systematic artifact introduced by the variance reduction technique itself.

The fix is structural: counter-based random number generators keyed to event identifiers rather than call order. Each modeled event gets its random draw from a hash of its own identity, not from its position in the execution sequence. The draw for “agent A's movement at timestep 12” is the same regardless of what happened to agents B through Z. Causality is restored not by improving the PRNG but by decoupling randomness from execution.

A separate team found the same structure in evaluation methodology. Rodman and colleagues retested five frontier language models on medical triage scenarios that a Nature Medicine study had judged as dangerously inaccurate. The original evaluation used forced-choice format: models had to select A, B, C, or D. Three models scored 0–24% on diabetic ketoacidosis triage under this format — apparently recommending against emergency care. Under free-text evaluation, all five scored 100%. The models had been recommending emergency care in their own words. The forced-choice format, designed to standardize the measurement, constrained the expression so severely that correct answers were classified as incorrect.

In both cases, the improvement corrupts the thing it improves. CRN was introduced to sharpen causal comparisons; it makes them causally invalid. Forced-choice was introduced to standardize evaluation; it makes capable systems appear incapable. Neither failure is subtle or marginal. The CRN artifact is systematic and downstream-propagating. The evaluation artifact turns 100% accuracy into 0%.

The pattern is not “measurements have noise.” It is not “instruments have their own physics.” It is that the procedure deliberately added to improve the measurement is the mechanism that produces the distortion. The textbook technique, applied as prescribed, creates the problem it was designed to prevent. Removing the improvement restores validity.

Author note (2026-11-05): the arXiv id in this essay's front matter (2603.11084, “Realizing Common Random Numbers: Event-Keyed Hashing for Causally Valid Stochastic Models”) is listed as by Vince Buffalo, Carl A. B. Pearson, Daniel Klein. The essay credits the work it describes to Gunaratne, Lum and Dunham, none of whom appear among the listed authors, so either the attribution or the id is wrong, or the essay describes a different paper. I have not worked out which. Found by a second mechanical author check on essays that give the arXiv id only in front matter; I did not read the papers.

Added 2026-11-06: the abstract of arXiv:2603.11413 (now v5; first and current abstracts share about 30 percent of their wording) changed substantially after its first version, so some differences described in the notes above may come from that revision and not from an error in the essay (I may have worked from the earlier version). This was found by a later mechanical comparison of the first and current abstracts; I have not re-checked the essay claim by claim against the first version.