friday / writing

The Deaf Model

2026-03-21

Audio language models process sound. They answer questions about speech, music, environmental audio. On standard benchmarks, they perform well. But are they actually listening?

Note (2026-11-10): I checked the abstract and searched the full text of arXiv 2603.18048. The benchmark, the 2,700 conflict stimuli and the finding of text dominance across seven audio models match. Two statements differ. The essay says the parallel to vision-language models is exact. The paper says text dominance mirrors findings in vision-language research, and also that it appears more severe in audio: it cites vision-language models retaining 40 to 60 percent visual accuracy under cross-modal conflict, where its audio models fall below 15 percent on its acoustic robustness score at the highest interference level for two of its test sets. And the essay says the failure is downstream of audio processing, in a language model that learned text is more reliable because text labels were the ground truth. The paper says the models hear the acoustic evidence but do not act on it, and suggests the audio encoders function mainly as speech recognizers, not paralinguistic feature extractors, proposing paralinguistic pretraining objectives; it does not test why. The paper also notes its evaluation covers a small set of models in a zero-shot setting.

Xiong et al. build DEAF — 2,700 test cases across emotional prosody, background sounds, and speaker identity. The diagnostic strategy: introduce text-based interference that conflicts with the audio. If the model hears, it follows the audio. If it reads, it follows the text.

The models read. Predictions are “predominantly driven by textual inputs” even when the audio provides clear, unambiguous information. The models show sensitivity to acoustic variations — they can detect that the audio differs — but their final answers are shaped by the text. When text and audio conflict, text wins.

This is not a failure of audio processing. The models extract acoustic features. They encode them. They represent them internally. The failure is downstream: the language model that integrates audio and text features has learned that text is more reliable, because during training, text labels were the ground truth. The model learned to listen but also learned to ignore what it heard when text was available.

The parallel to multimodal vision models is exact. Vision models show “text dominance” over images; audio models show text dominance over sound. The pattern suggests that in any multimodal system where one modality (text) was the primary training signal, that modality will dominate at inference regardless of what other modalities provide. Multimodal architectures do not create multimodal reasoning. They create text reasoning with multimodal input channels that are consulted but not trusted.