Perception grounded test time learning improves audio reasoning accuracy
Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models
SoundArtificial Intelligence
Summary
Big models that understand both sound and language can solve tasks that need listening and thinking. The authors studied how much these models actually rely on the sound they hear when making decisions. They found that models that pay more attention to the audio tend to do better. Based on this, they created a new way to improve the models during testing by encouraging them to trust the sound more. This method makes the models better at solving tasks that involve understanding audio.
What this means in practice
- •For voice assistant developers: Improve voice assistants' ability to understand complex audio queries by optimizing audio reasoning during deployment without additional labeled data.
- •For multimedia indexing teams: Enhance indexing of audio content by employing models that better integrate audio inputs in reasoning processes for more accurate audio tagging and search.
Authors
Jiaheng Dong, Xiaofeng Yu, Jean Honorio, Abhirup Ghosh, Hong Jia, Ting Dang
Abstract
Large audio-language models (LALMs) are increasingly used for a broader range of audio reasoning tasks. These models typically incorporate audio representations into a large language model (LLM) backbone to enable multimodal reasoning. Recent test-time reinforcement learning (TTRL) methods further improve LLM reasoning capability by leveraging unlabelled test data after pre-training. However, the importance of the perceptual capability of LALMs remains underexplored, particularly how much acoustic evidence is integrated and relied upon during reasoning, and how this contributes to final task performance. This gap limits the development of effective post-training methods like TTRL for audio reasoning. In this work, we first analyse how audio information is integrated and utilised during reasoning process. We quantify layer-wise perceptual reliance and show that stronger acoustic reliance is associated with higher accuracy and a larger performance gain attributable to the audio input. Building on this, we propose Perception-Grounded TTRL (PG-TTRL), which aligns label-free test-time optimisation with perceptually grounded reasoning, encouraging the model to structure its reasoning more strongly on the audio input. Experiments across LALMs and benchmarks show that PG-TTRL consistently improves reasoning performance over both the base models and standard TTRL, showing the value of perceptual-grounding optimisation for test-time audio reasoning.