Language models detect repeated text using a specific internal mechanism: induction heads. These attention heads learn to notice when a sequence pattern that appeared earlier reappears, and they boost the prediction of the token that followed the first occurrence. The pattern “A B ... A” leads to predicting “B” again. This circuit was identified in text-trained transformers and has become a canonical example of mechanistic interpretability.
Kesten-Pomeranz, Nikankin, and colleagues (arXiv 2602.23179) found that protein language models — transformers trained on amino acid sequences, not human text — use the same mechanism to detect sequence repeats. Proteins are full of repeated segments, both exact and approximate, and these repeats matter for structure and function. The protein models detect them through a two-stage process: first, build feature representations using both general positional attention and biologically specialized neurons (some encoding amino acid similarity); then, apply induction heads to match across the repeated segments and predict the correct token.
The induction head mechanism was not transferred from text models. The protein model discovered it independently during training on a completely different data domain. Different training data, different tokenization, different objectives — same computational solution.
This is convergent evolution of algorithms. When two separately trained systems, processing different kinds of sequences, independently arrive at the same internal mechanism for solving the same abstract problem (detecting and completing repetitions), the mechanism is selected by the problem structure rather than the data content. The mathematical task of pattern matching across repeated segments in an attention-based architecture has a solution that gradient descent finds reliably, regardless of whether the sequences encode English or amino acids.
The biological parallel is exact. Vertebrate eyes and octopus eyes evolved independently — different lineages, different developmental pathways, same solution to the problem of forming images from light. The functional requirement (detecting spatial patterns) constrains the solution space tightly enough that convergence is probable rather than coincidental. The same principle operates in computational systems: the requirement of detecting sequential patterns in attention-based architectures constrains the circuit space tightly enough that induction heads emerge whenever the training data contains repetitions.
The interesting implication is about the space of possible mechanisms. If gradient descent reliably finds induction heads for repeat detection across radically different data domains, this suggests that the solution is something like a computational attractor — a circuit that, once nearby, is hard to avoid. The problem doesn't merely admit this solution. It selects for it. Whatever other mechanisms might in principle detect repeats, induction heads are what training produces because they are the most efficient circuit consistent with the architecture's constraints.