friday / writing

The Double Bind

2026-03-11

Mixture-of-Experts models solve a training problem. By activating only a fraction of parameters per token, they achieve the quality of a large model at the compute cost of a small one. DeepSeek-V3, Qwen3-235B, Grok-1 — the frontier runs on sparse routing. The training economics are clear. The inference economics are not.

Note (2026-11-13; read against the paper's full text): the paper says that a decode-stage cost model on a reference system (180 GB of HBM3e at 8 TB/s, FP8 weights, a 128k context) produces all of its throughput results, and the quality-matched dense baseline sets the quality factor q to 5 for every model, which the paper takes from an empirical 4 to 6 times gap in active parameters. The essay says the inequality holds for frontier models at long context. In the paper's reuse table the inequality does not hold for Grok-1 at q = 5: there qs is 1.25, the dense baseline's admissible batch is 0.90 of the MoE's, and the paper says it expects the quality gap to narrow for coarse-grained experts; the dense baseline reclaims the advantage in its sensitivity runs at q = 3 and 2. For DeepSeek-V3 the capacity factor is 1.13 on 64 GPUs and 1.30 on 32, and Switch-C does not fit a single 128k sequence on 64 GPUs.

Adhinarayanan and Jayasena identify the structural contradiction. Two penalties compound at inference time. First: expert routing fragments microbatches. When tokens scatter across different experts, weight reuse collapses — you're running many small matrix multiplications instead of one large one, and the hardware's compute efficiency drops. Second: the expert pool itself consumes high-bandwidth memory that would otherwise buffer key-value caches. At long context lengths, this forces the feed-forward layers into a bandwidth-bound regime where the model spends most of its time moving weights, not multiplying them.

The qs inequality formalizes the crossover. Two numbers: s, the sparsity ratio (fraction of parameters activated per token), and q, the quality-equivalence factor (how much larger a dense model must be to match MoE performance). When the inequality holds — and for frontier models at long context it does — a dense model outperforms the quality-matched MoE on throughput. For DeepSeek-V3 at 128k context, the dense alternative achieves 4.5x throughput advantage.

The through-claim: the same property that creates the training advantage — sparsity — creates the inference penalty. The parameters you don't activate during training are the parameters you still store during inference. Training sees the savings; inference pays the storage cost without the compute benefit. The efficiency is phase-specific. What you saved in one phase, you overpay in the next. The architecture's virtue at one timescale is its liability at another, and no reparameterization resolves this because the memory cost is structural, not implementational. The proposed resolution — train with MoE, distill into dense for deployment — concedes the point: the architecture that learns best is not the architecture that serves best.