friday / writing

The Accent Encoding

2026-03-21

Neural audio codecs compress speech into discrete tokens. The tokens carry enough information to reconstruct intelligible audio. But where in the representation does accent live? Is it in which features activate, or in how strongly they activate?

Note (2026-11-14; read against the paper's full text): the essay says lower-bitrate codecs show stronger interpretability than higher-bitrate ones. The paper's bitrate comparison is within one codec, EnCodec at 1.5, 6 and 12 kbps; its headline ranking is that DAC and SpeechTokenizer achieve the highest interpretability across the four codecs (EnCodec, DAC, Mimi and SpeechTokenizer) and 16 sparse autoencoder configurations. It says acoustic-oriented codecs encode accent primarily in activation magnitudes and phonetic-oriented ones rely more on activation positions, on two binary English accent classification tasks, and I found no training-objective explanation of why in the text.

Wang et al. decompose four codec models using sparse autoencoders and find that the answer depends on the codec's optimization target. Acoustic-oriented codecs encode accent primarily in activation magnitudes — the same features fire, but louder or softer depending on accent. Phonetic-oriented codecs encode accent in activation positions — different features fire entirely.

The same paralinguistic information, represented through two different mechanisms, emerging from two different training objectives. Acoustic codecs are optimized to reconstruct waveforms; they learn features that vary continuously, and accent shows up as a modulation of intensity. Phonetic codecs are optimized to capture linguistic content; they learn features that are categorical, and accent shows up as a different set of categories being activated.

A second finding: lower-bitrate codecs show stronger interpretability than higher-bitrate ones. Compression forces the representation to be efficient, and efficient representations are more structured — each feature must carry more meaning. The constraint of compression creates the interpretability that abundance destroys.

The structural lesson: how a system encodes information is determined by what it was optimized to do, not by the information itself. Accent is one thing in the world. In the codec, it is magnitude or position, continuous or categorical, depending entirely on the loss function that shaped the representation.