Large audio language models struggle to combine capabilities accurately
When Capabilities Fail to Compose: Diagnosing the Compositionality Gap in Large Audio-Language Models
Sound
Summary
Large audio-language models can do separate tasks well, like recognizing sounds or answering questions from audio. However, this study found that when the models try to combine these tasks—for example, recognizing sound attributes while answering questions—their performance drops significantly. The authors tested this with different types of sounds and questions and found consistent errors. This shows that current models still have trouble juggling multiple audio understanding skills at once.
What this means in practice
- •For voice assistant developers: Identify the limits of combining audio recognition and question answering to improve multi-step voice interaction reliability.
- •For audio software engineers: Diagnose and refine audio processing pipelines to address failure points when multiple audio attributes need joint interpretation.
Authors
Chien-Feng Liu, Chih-Kai Yang, Bo-Han Feng, Yu-Hsuan Li Liang, Hung-yi Lee, Cheng-Fu Chou
Abstract
Large audio-language models (LALMs) perform strongly on individual audio tasks, but whether these capabilities can be reliably composed remains underexplored. We conduct a controlled diagnostic study of capability composition in LALMs, requiring models to integrate audio-attribute recognition, cue-conditioned segment selection, and downstream ASR or question answering. We construct two-utterance inputs with distinct acoustic cues to evaluate composition across environmental sound, gender, and emotion cues, with ASR, Math QA, and Factual QA as downstream tasks. Across four open-source LALMs, compositional QA accuracy decreases in 39 of 40 model-task-cue settings, by an average of 26.7 percentage points. ASR exhibits a similarly consistent degradation, with WER increasing in 39 of 40 settings by an average of 28.5 percentage points, while the magnitude of degradation varies across models, cue types, and cue salience. We further probe these failures through output format, positional preference, and chain-of-thought (CoT) analyses. Our study reveals a systematic gap between possessing individual audio capabilities and reliably composing them.