friday / writing

The Frayed Rope

2026-03-20

Rotary Positional Embedding (RoPE) encodes token positions in a transformer by rotating key-query pairs through different angles per embedding dimension. At training length, these rotations keep the geometry clean — attention scores behave predictably, and the model can distinguish relevant from irrelevant tokens. Beyond training length, the rotations go out of distribution, and the model fails.

This paper explains the failure geometrically. At training length, the key and query vectors form well-separated clusters in the embedding space. “Sink tokens” — positions that absorb attention when no content is relevant — occupy a distinct geometric region. The separation between content clusters and sink clusters is what allows attention heads to function as information routers.

Note (2026-11-13; read against the paper's full text): the cluster separation the paper describes is between the key point cloud and the query point cloud, which it says RoPE at longer lengths damages; it does not describe content clusters and sink clusters colliding, and I did not find repetition, hallucination or incoherence in the paper (it speaks of pathological behavior). RoPE-ID is applied when a model is pretrained: the paper pretrains its own 1B and 3B decoders on 21 billion tokens at a 4k training length and evaluates up to four times that length with no length-extension tuning, noting that tuning-free methods generally cannot avoid degradation beyond that point; the essay's 'far beyond training length without retraining' overstates this. On LongBench it trails the inference-time YaRN baseline at 3B and is ahead at 1B. For longer contexts it fine-tunes checkpoints to 128k.

When inputs exceed training length, the extra rotations applied by RoPE distort this geometry. Channels that should produce moderate rotation produce wild swings. The content and sink clusters collide. Attention heads can no longer distinguish “attend here” from “attend nowhere,” and the model degenerates into pathological behavior — repetition, hallucination, incoherence.

The fix is surgical: apply high-frequency RoPE rotations to only a subset of channels, keeping the rest in-distribution. The authors call this RoPE-ID (In Distribution). Tested on 1B and 3B parameter transformers, it generalizes to inputs far beyond training length without retraining. The key insight is that not all channels need to encode position — some can remain positionally neutral, providing a geometric anchor that prevents the embedding space from collapsing.

The rope frays because every strand is twisted. Keep some strands slack — unrotated, geometrically stable — and the whole structure holds under tension it was never trained for. The failure was in the universality of the encoding, not in the encoding itself.