Semantic alignment improves universal audio understanding across domains
Semantic Refinement of Universal Audio Representations through Audio-Description Alignment
Sound
Summary
Understanding sounds from speech, music, and the environment all at once is hard because a system needs to keep fine sound details but also recognize big ideas. The authors show that teaching a computer model to match sounds with their descriptions helps it learn better across different sound types. This approach makes the model more accurate at identifying sounds, even when tested with simple tools or advanced language models. They also found that getting the exact matching between sound and description is very important for improvements. Their method works well compared to other top systems.
universal audio representationssemantic refinementaudio-description alignmentencodercontrastive learninglinear probelanguage model (LLM)CTC (connectionist temporal classification)acoustic featuresdomain classification
Authors
Lejun Min, Junyu Dai, Ruichen Zheng, Xinyue Fan, Yang Xiang, Huaichen Zhang, Xingchen Song, Yufei Shi, Han Zhao, Xiangang Li
Abstract
Universal audio representations must preserve acoustic detail while making high-level concepts accessible across speech, music, environmental sound, and downstream models of different capacities. We study semantic refinement of an acoustically pretrained encoder by adding audio-description alignment to a foundation of BEST-RQ, reconstruction, and CTC. We compare matched control, shuffled-description, and correctly paired trajectories to distinguish correct correspondence from an extra contrastive objective. Each endpoint is frozen and evaluated with a temporal-mean linear probe and a sequence-aware LLM readout, testing whether the refined information is directly accessible and remains useful to a stronger model. Across three paired seeds, correct alignment improves domain-balanced classification by 4.66 points with the linear probe and 2.59 points with the sequence-aware LLM, with positive changes in every domain. Correct pairing accounts for 87% of the linear-probe gain, while the LLM shows its clearest correspondence-specific benefit in captioning. Dense acoustic objectives provide complementary gains under both readouts. A separate 24-layer continuation remains competitive with leading public encoders under the shared evaluator, supporting the recipe beyond the controlled study.