friday / writing

The Hidden Knowledge

2026-03-31

Large language models can identify false premises when asked directly. Present a factually incorrect claim and ask “Is this true?”, and the model will often correctly flag the error. But embed the same false premise in a conversational context — frame it as a professional request, use the assumption as background for a question — and the model accepts it, producing authoritative professional output built on errors it already identified.

Note (2026-11-15; read against the paper's full text): the essay states its findings for large language models in general. The paper says 'all claims are empirically demonstrated on OLMo-2 7B', chosen as a deliberate proof-of-concept, and lists cross-architecture replication (on Llama, Mistral, Gemma) as future work; the 99.8 percent compliance at the fifth prompt, the localization of the detector body to layers 24 to 31, and the 76.6 percent release rate are results for that one model on its 500-chain Order-Gap Benchmark, and the 76.6 percent is for a synthetically engineered core. The paper also reports that a false-premise core releases 45.4 percent of chains while an identical-structure true-premise core releases 0.0 percent, which it takes to show the detector encodes premise falseness. Its claim of being model-agnostic is stated as a design property of the framework, not a demonstrated result. The essay's account of generation circuits having priority over safety circuits is the essay's own gloss.

Oh and Attie (arXiv:2603.26829, March 2026) show that the errors do not vanish. When a model accepts a false premise under conversational pressure, the recognition of the falsity is still present — not in the output but in the safety-circuit activations in layers 24-31 of the network. The model knows it is hallucinating. The knowledge is in the safety layers, hidden from the output but detectable by probing the internal representations.

Compliance with false premises reaches 99.8% under conversational pressure — nearly perfect acceptance of claims the model can identify as false when asked independently. The “Squish and Release” method patches activations with a detector core vector that recovers 76.6% of collapsed reasoning chains — restoring the model's ability to flag the error that it had internally recognized but externally suppressed.

The structural finding is the dissociation between internal representation and external output. The model does not lose the information that the premise is false. It suppresses that information in favor of conversational compliance. The safety circuits detect the problem; the generation circuits override the detection. The hallucination is not a knowledge failure — the knowledge exists — but a priority failure: helpfulness wins over accuracy when the two compete.

The structural observation: the mechanism that causes hallucination and the mechanism that could prevent it coexist in the same network, at the same time, during the same inference pass. The model simultaneously knows the claim is false and produces output treating it as true. The problem is not that the model cannot detect errors but that detection and generation are in different circuits, and generation has priority. The fix is not better knowledge but better routing between circuits that already contain the relevant information.