Large audio language models mainly learn text tricks after post-training
What Does LALM Post-training Learn? Rethinking Audio Understanding and Reasoning Capabilities
SoundComputation and LanguageMultimedia
Summary
Improving large audio-language models through extra training often seems to help them understand sounds and answer questions better. The authors found that much of this improvement actually comes from learning to spot patterns in the type of questions asked, rather than truly understanding the audio. Their tests showed the models still performed well even when the audio was removed, but struggled when the question format changed. This means these models mostly learn to guess answers based on question styles instead of getting better at really understanding or reasoning about audio.
What this means in practice
- •For audio model developers: Design evaluation methods that separate true audio understanding from question format tricks in model training.
- •For voice interface designers: Avoid relying on model improvements that only work for specific question types when building robust audio question answering systems.
Authors
Hua-Hsuan Chen, Xuanjun Chen, Wei-Chung Lu, Chih-Kai Yang, Yu-Xiang Lin, Sung-Feng Huang, Haibin Wu, Hung-yi Lee
Abstract
Post-training often improves the performance of large audio-language models on audio understanding and reasoning benchmarks. However, these improvements may reflect adaptation to specific question formats, such as multiple-choice questions, rather than genuine advances in audio understanding and reasoning capabilities. To investigate what LALMs actually learn from post-training, we conduct controlled experiments that disentangle improvements in audio understanding from adaptation to specific question formats. Across different models and post-training methods, we find that much of the reported improvement persists even without informative audio, but largely disappears when the question format changes. These findings suggest that post-training often teaches models to exploit task-specific textual cues and answer patterns rather than developing generalizable audio understanding and reasoning capabilities.