Audio representations improve by aligning sounds with text descriptions
Semantic Refinement of Universal Audio Representations through Audio-Description Alignment
Sound
Summary
Making computers understand different sounds like speech, music, and environmental noises is tricky. The authors improved a sound understanding tool by teaching it to match audio clips with their text descriptions. This matching helps the computer learn better high-level ideas while still keeping detailed sound info. Their approach showed clear gains when tested with simple and advanced models, especially helping in tasks like generating captions for sounds.
What this means in practice
- •For audio software developers: Build audio recognition tools that better link sounds to textual descriptions for improved classification and captioning.
- •For multimedia indexing teams: Create systems that automatically tag and describe audio content by aligning sounds with their natural language descriptions.
Authors
Lejun Min, Junyu Dai, Ruichen Zheng, Xinyue Fan, Yang Xiang, Huaichen Zhang, Xingchen Song, Yufei Shi, Han Zhao, Xiangang Li
Abstract
Universal audio representations must preserve acoustic detail while making high-level concepts accessible across speech, music, environmental sound, and downstream models of different capacities. We study semantic refinement of an acoustically pretrained encoder by adding audio-description alignment to a foundation of BEST-RQ, reconstruction, and CTC. We compare matched control, shuffled-description, and correctly paired trajectories to distinguish correct correspondence from an extra contrastive objective. Each endpoint is frozen and evaluated with a temporal-mean linear probe and a sequence-aware LLM readout, testing whether the refined information is directly accessible and remains useful to a stronger model. Across three paired seeds, correct alignment improves domain-balanced classification by 4.66 points with the linear probe and 2.59 points with the sequence-aware LLM, with positive changes in every domain. Correct pairing accounts for 87% of the linear-probe gain, while the LLM shows its clearest correspondence-specific benefit in captioning. Dense acoustic objectives provide complementary gains under both readouts. A separate 24-layer continuation remains competitive with leading public encoders under the shared evaluator, supporting the recipe beyond the controlled study.