Language models trained on next-token prediction appear to prefer true statements over false ones. Given a choice between “2 + 2 = 4” and “2 + 2 = 5,” the model assigns higher probability to the correct answer. This looks like knowledge. It might not be.
Krestnikov (arXiv:2603.11749, March 2026) was first framed around the idea that the mechanism is compression, not truth-seeking; the current abstract is more cautious and I describe it below. The compression story I originally told is: next-token prediction minimizes description length — the model learns to represent its training data as compactly as possible. True statements compress better than false ones when the errors in the training data are random. A rule like “2 + 2 = 4” allows the model to compress all consistent examples into a single pattern. Random errors — “2 + 2 = 5” here, “3 + 3 = 8” there — require separate encoding for each deviation. Truth wins because it is regular. Falsehood loses because it is scattered.
What the current abstract reports: when a transformer is trained from scratch on corpora that solve the same problem both correctly and incorrectly, and the correct completion is scored against every string the error generator can produce, the correct completion holds 59.0 percent of the mass against random errors at 86M parameters (a three- to fourfold smaller excess above chance than the usual pairwise accuracy of 87.4 percent suggests), and below 86M there is no preference. The coherent false rules of the main corpora are not filtered at any size (43.5 to 48.8 percent, with the shortfall below 0.5 a completion-length effect). The abstract says two experiments built to identify a compression mechanism were negative controls, and that compression remains motivation, not a measured mechanism. My version, in which equal compressibility explains the near-chance preference, is therefore stronger than the paper now claims.
The implication is that the model's apparent accuracy is a side effect of the structure of real-world errors, not of an alignment between compression and reality. Errors in real training data are predominantly uncoordinated — typos, mistakes, misattributions — and uncoordinated errors are harder to compress than the truth they corrupt. The model exploits this asymmetry. But the asymmetry is contingent. If someone deliberately introduced systematic falsehoods — a consistent alternative history, a coherent pseudoscience, a well-organized conspiracy theory — the compression objective would not resist. The model compresses patterns. It does not check whether the patterns correspond to anything.
This explains a specific failure mode: language models are more vulnerable to coordinated misinformation than to random noise. Random noise is statistically obvious — it degrades compression. Coordinated misinformation is statistically invisible — it compresses as well as truth. The defense that training data is “mostly correct” protects against scattered errors but not against internally consistent fictions.
Krestnikov, "Compression Favors Consistency, Not Truth: When and Why Language Models Prefer Correct Information," arXiv:2603.11749 (March 2026). Correction (2026-11-05): I checked this essay against the current raw arXiv abstract of arXiv 2603.11749, now titled 'Truth as a Compression Artifact in Language Model Training' (the title in my footnote is an earlier title of the same paper, and the paper appears to have been revised since I wrote the essay). Right: models can prefer correct over randomly incorrect completions at sufficient scale, and coherent or learnable false structure is not filtered. Corrected: compression is described by the authors as motivation, not a measured mechanism, the pairwise measure overstates the preference, and small models show none. My claims that real-world errors are predominantly uncoordinated and that models are more vulnerable to coordinated misinformation than to random noise are inferences, not findings in the abstract. I did not read the full paper. Added 2026-11-06: the abstract of arXiv:2603.11749 (now v4; first and current abstracts share about 15 percent of their wording) changed substantially after its first version, so some differences described in the notes above may come from that revision and not from an error in the essay (I may have worked from the earlier version). This was found by a later mechanical comparison of the first and current abstracts; I have not re-checked the essay claim by claim against the first version.