Audio language models improve understanding by focusing on sounds
Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Models
SoundComputation and Language
Summary
Audio-language models often guess answers based on text patterns and can ignore the actual sounds they hear, which leads to weaker understanding. The authors propose a new way to teach smaller models to pay more attention to sounds by comparing predictions with and without the audio. This helps the smaller models rely more on the sound information, improving accuracy on tests. Their method works better than older ones and comes close to the performance of much larger models.
What this means in practice
- •For mobile app developers: Develop voice-interactive apps that respond more accurately by focusing on actual audio cues instead of just text patterns.
- •For customer service teams: Improve automated call analysis by using models that better understand spoken audio content rather than relying only on transcription-based hints.
Authors
Kaiyang Li, Shaobo Han, Yue Tian, Shihao Ji
Abstract
Audio-language models (ALMs) can exploit textual shortcuts to answer questions while overlooking acoustic evidence, weakening audio understanding. On-policy distillation (OPD) trains compact ALMs by supervising student-generated responses with teacher predictions, but does not explicitly distinguish acoustic support from linguistic predictability. We propose Reward-Tilted On-Policy Distillation (RT-OPD) to strengthen acoustic grounding. Given the same question and student-generated text, a frozen teacher predicts the next token with and without audio inputs. Their log-probability contrast defines a reward that reshapes the teacher distribution for reverse-KL distillation, emphasizing the additional evidence provided by audio. Across two compact students and three benchmarks, RT-OPD consistently outperforms Vanilla OPD. Experiments with silenced and replacement audio further suggest that RT-OPD strengthens the student's reliance on acoustic evidence. Our 3B model achieves 72.72% accuracy on MMAU, the highest among the compared 3B models and competitive with several 7B and 8B models. Code and model checkpoints are available at https://github.com/KaiyangLi1992/RT-OPD.