Text-to-audio generation has a vocabulary problem. The word “crash” covers a ceramic plate, a metal tray, a glass window, and a car wreck — acoustically unrelated sounds sharing a label. Existing video-to-audio systems condition on text captions, inheriting this ambiguity. The generated sounds are semantically correct and acoustically vague.
Note (2026-11-13; read against the paper's full text): the paper's model does not bypass text. It conditions on three inputs, the video, a reference audio clip and a text prompt describing semantic requirements, written as G(V, Ac, T) in its problem formulation, and it follows MMAudio in training on both audio-text-visual and audio-text datasets, 'unifying video, audio and text modalities'; in its evaluation the reference audio is the last 2 seconds of the original clip and the category name serves as the text prompt. I did not find a text-free mode in the text I searched. What the paper says is that text alone leaves two bottlenecks (coarse labels in training data, and ambiguity of text for fine acoustic features), and that adding reference audio gives finer control; its examples are timbre transfer (a violin melody to a suona), footsteps on different surfaces, and a zero-shot silenced gunshot, where it says a text prompt fails. So the essay's 'the text caption is unnecessary' and 'the bottleneck was text' go beyond the paper, and the paper also names a synchronisation difficulty (simply overlaying the reference sound on the footage causes problems) that its method is designed to address. The Foley-artist analogy and the claim that audio is its own best description are the essay's.
Fang et al. bypass text entirely. AC-Foley conditions audio generation on a reference audio clip rather than a text description. Show the system a video of something hitting a metal surface and give it the sound of a different metal impact — it generates the correct synchronization with the timbre of the reference. The text caption is unnecessary.
The through-claim: audio is its own best description. A text prompt compresses acoustic detail into categorical labels. A reference audio sample preserves the exact spectral characteristics, temporal envelope, and timbral quality that text discards. Conditioning on audio rather than text doesn't just add precision — it removes a lossy compression step from the pipeline.
The capabilities this unlocks — timbre transfer, zero-shot generation for sounds the model hasn't seen labeled, fine-grained control over acoustic attributes — are consequences of removing the bottleneck, not new features built on top. The bottleneck was text.
This is the Foley artist's method made computational. A Foley artist doesn't describe the sound they want in words. They find a reference — a coconut for hooves, a celery stalk for breaking bones — and match the recording to the picture. AC-Foley does the same: the reference IS the specification.