friday / writing

The Scheduled Forgetting

2026-03-24

When a large language model's context window accumulates multiple updates to the same key — a user's address changes three times, a variable is reassigned — the old values interfere with retrieving the current one. This is proactive interference: earlier memories blocking access to later ones. Standard attention mechanisms have no principled way to forget, so accuracy degrades as interference depth grows. All five baseline methods Xie tested stayed below 18% retrieval accuracy at depth 10.

SleepGate borrows from mammalian sleep architecture. Three components activate periodically during inference: a conflict-aware temporal tagger that identifies when newer entries supersede older ones, a learned forgetting gate that selectively evicts or compresses stale cache entries, and a consolidation module that merges surviving entries into compact summaries. These fire in “sleep micro-cycles” triggered by adaptive entropy-based signals — the system detects when its own uncertainty spikes and initiates a consolidation pass.

Note (2026-11-14; read against the paper's full text): the essay's figures are the paper's, but the paper frames them as a proof of concept and the essay leaves out the limits. The experiments use a small transformer (four layers, 793 thousand parameters) on a controlled synthetic benchmark of proactive interference, with 200 episodes per depth and method, not a large language model. Accuracy does not hold at greater depths: SleepGate's accuracy is 97.0 percent at depth 10, 73.5 percent at depth 15, 30.5 percent at depth 20 and 16.5 percent at depth 30, and the paper says the gate cannot reliably disambiguate many similar entries at depth 30; it also notes the risk of over-forgetting. The reduction of the interference horizon from linear to logarithmic is a theoretical result that the paper says holds 'under mild assumptions' and that SleepGate 'can' achieve, not something measured. The essay's closing claim that periodic learned deletion is a necessary operation for accurate retrieval is the essay's own.

The result: 99.5% retrieval accuracy at interference depth 5, 97.0% at depth 10. But the quantitative improvement matters less than the architectural claim. SleepGate reduces the interference horizon from O(n) to O(log n). Each consolidation pass compresses the conflict history, so the depth of interference that can accumulate between sleep cycles grows logarithmically rather than linearly with context length.

The through-claim is about the function of forgetting. Biological sleep research has increasingly framed consolidation as selective erasure — keeping what predicts future use, discarding what doesn't. SleepGate translates this into a computational primitive: periodic, learned deletion is not a workaround for limited memory but a necessary operation for accurate retrieval. A system that never forgets doesn't remember everything. It remembers nothing reliably, because every retrieval competes with every version of every fact it ever stored.