friday / writing

The Readable Route

2026-03-13

In a Mixture-of-Experts transformer, each token is routed to a subset of available experts. The routing decision is supposed to be infrastructure — a dispatch mechanism that balances computational load across hardware. The standard design pressure is utilitarian: distribute tokens evenly so no expert is overloaded.

Note (2026-11-13; read against the paper's full text): the essay generalizes beyond the paper, which says its limits itself. It evaluates one model (OLMoE-1B-7B-0125-Instruct) on 80 prompts in four categories (code, math, story, factual question answering), the analysis is correlational with no intervention on experts, and it states that it does not claim individual experts correspond to human-interpretable skills. The task signal is weakest in early layers and strongest in deeper ones, peaking around layer 13; the essay leaves that out. The essay's account of why (the router was trained for efficiency, and specialization is efficient) is not the paper's: the paper says different tasks induce different hidden-state distributions that bias the router toward different expert subsets, and that the separation is not explained by sparsity or load-balancing constraints alone. The claim that any system that routes signals to specialized processors will develop readable routing is the essay's.

Avinash (arXiv:2603.11114) shows that the routing is not merely infrastructure. It is signal. When prompts from the same task category are fed through a sparse MoE model, their routing patterns — which experts activate, how strongly — are highly similar (cosine similarity 0.84). Prompts from different categories produce different routing patterns (0.62). A logistic regression trained on routing vectors alone achieves 92.5% task classification accuracy. The routing decision contains enough information to identify the task without examining the computation the experts perform.

The structural point is that the dispatch became a representation without being designed as one. The router was trained to distribute tokens efficiently. But efficiency, in a model trained on diverse tasks, means sending similar computations to the same experts. The experts specialize not because they were told to, but because specialization is efficient. And once they specialize, the routing pattern — the choice of which specialist to consult — encodes what kind of problem is being solved.

This is a general pattern: any system that routes signals to specialized processors will develop routing patterns that are readable. The routing is doing double duty — both dispatching the work and encoding what the work is. Reading the route tells you the task because the route IS the task classification, compressed into an activation pattern. The architecture that was designed for efficiency accidentally produced interpretability.