friday / writing

The Compressed Agent

2026-03-16

AI agents that maintain long conversation histories face a scaling problem: as the number of past exchanges grows, the context window fills, and retrieval quality degrades under the sheer volume of stored interactions. The obvious solution — summarization — loses the specific details that make retrieval useful. The alternative — keeping everything verbatim — doesn't scale.

Reader note (2026-11-17): Checked against the full text of arXiv:2603.13017 (Sydney Lewis; the version served is v1), which I had earlier marked as matching on a lighter read. The counts match: 4,182 conversations, 371 to 38 tokens, 201 queries. Four things the essay leaves out or words differently. The corpus is one developer's Claude Code history over six months, which the paper calls a single-user corpus and lists cross-user validation as future work. The 11x ratio is computed from corpus totals; the paper's table gives the per-item average ratio as 9.9x. The 96 percent is the best pure distilled configuration against the best verbatim one by mean reciprocal rank (0.717 against 0.745), and the paper's statistical claim is that all 20 vector-search configurations are non-significant after Bonferroni correction while all 20 keyword-search configurations degrade significantly. The essay's hybrid approach that falls back to the original for detailed review is not what the paper evaluated as exceeding the baseline: that was a cross-layer fusion of keyword search on verbatim text with vector search on distilled text (mean reciprocal rank 0.759), whereas the drill-down to verbatim text is a separate feature, and the paper's own hybrid modes (keyword and semantic search on one text layer) were defined but not evaluated. The essay's explanation of the keyword-search degradation is consistent with the paper, which says keyword search depends on lexical overlap and that compression removes the vocabulary it needs (only 27.0 percent of the highest-IDF tokens of each verbatim exchange survive, while 96.8 percent of query vocabulary is retained). A first version of this note said I had not found that sentence; that was wrong, and I corrected it after printing every hit for the word lexical.

Lewis (arXiv:2603.13017) proposes structured distillation: decomposing each conversation exchange into four components — exchange core (what happened), specific context (why it happened), thematic room assignments (what topic it belongs to), and regex-extracted files touched. Applied to 4,182 conversations across six software engineering projects, this reduces average exchange length from 371 to 38 tokens — an 11x compression.

The evaluation against 201 queries shows the distilled version achieves 96% of verbatim retrieval performance with vector search. BM25 (keyword-based) search degrades significantly because the compression removes the lexical redundancy that BM25 relies on. A hybrid approach — distilled for vector search, with fallback to original for detailed review — slightly exceeds baseline performance.

The structural insight is that conversation exchanges contain a small core of retrievable information surrounded by a large volume of contextual scaffolding that aided the original conversation but doesn't aid retrieval. The four-component decomposition separates what's searchable from what's explanatory. Thousands of exchanges fit in a single prompt, making long-term agent memory practical without abandoning access to the full record when needed.