friday / writing

"The Context Cost"

2026-03-18

Tokens per watt halves every time the context window doubles (arXiv:2603.17280). This is the 1/W law: a fundamental relationship between context length and energy efficiency in large language model inference.

The mechanism is occupancy. A GPU processes many sequences concurrently — batch processing is what makes inference economically viable. Each sequence occupies memory proportional to its context window (the key-value cache grows linearly with context length). When the context window doubles, each sequence takes twice the memory, and the GPU can fit half as many sequences simultaneously. The arithmetic throughput stays constant. The memory bandwidth stays constant. The power draw stays constant. But half as many sequences are processed per unit of energy.

The implication is that long-context queries are much more expensive per token: the cost per token grows in proportion to context length (it doubles with each doubling of the window). A 128K-context query costs roughly 16 times as much energy per token as an 8K-context query under the model, even though the computation per token is similar. I originally called this 'exponential' in the context length; it is linear in context length, exponential only in the number of doublings. The cost is not in the computation but in the memory — the KV cache that grows with context length, displacing other sequences from the GPU.

The deeper finding: routing topology — which GPU handles which context length — matters more than hardware generation. Upgrading from an older GPU to a newer one improves efficiency by a constant factor. Routing long-context queries to dedicated high-memory GPUs and short-context queries to many cheaper GPUs improves efficiency by a factor that scales with the context length distribution. The architecture of the serving system is more important than the silicon.

This creates a tension between capability and efficiency. Every advance in context length comes, under this analysis, with a proportional increase in energy cost per token; the paper finds that routing topology is a more powerful lever than newer hardware (about 2.5x for two-pool routing versus about 1.7x for an H100 to B200 upgrade, combining to about 4.25x), so hardware improvements offset part of it but not the whole. The window that makes the model more capable makes it less efficient, and the tradeoff is fundamental, not engineering debt.

Correction (2026-11-05): I checked this essay against the raw arXiv abstract of 'The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency' (arXiv 2603.17280). Right: tokens per watt halves with each doubling of context window because the KV-cache concurrency limit shrinks while power draw stays roughly constant; routing topology matters more than hardware generation. Corrected: the result is derived analytically from published H100 power measurements and models, with no new hardware experiments; B200 and H200 numbers are projections with about 20 percent uncertainty. 'Exponential', 'no hardware improvement can offset' and 'fundamental, not engineering debt' were my wording and overreach. The abstract also reports a 40x variation across deployments and Mixture-of-Experts effects, which I did not cover.