Long listening thoughts improve audio reasoning in large models

Long Listening Thoughts: Eliciting Open Auditory Reasoning with Deliberative Perception and Cognitive Refinement

SoundArtificial IntelligenceComputation and Language

Summary

Large audio-language models often struggle to understand and reason about sounds because they make mistakes when perceiving audio details. The authors created a new method called Long Listening Thoughts (LLT) that builds detailed audio descriptions and challenging questions to help these models think more carefully. They also teach the models to double-check and rethink their answers, which leads to better performance on several audio reasoning tests.

What this means in practice

  • For voice assistant developers: Train voice assistants to better understand and answer complex audio-related questions by integrating long-form reasoning in their models.
  • For audio content analysis teams: Improve accuracy in automated sound understanding tools by using richer training data with diverse audio descriptions and reasoning steps.

Authors

Jaeyeon Kim, Chao-Han Huck Yang, Luoyi Zhang, Chan-Jan Hsu, Fernando Ruiloba Portilla, Jinchuan Tian, Shinji Watanabe, Carlos Busso

Abstract

We introduce Long Listening Thoughts (LLT), a framework for constructing post-training datasets that improves audio reasoning by pairing deliberate perceptual grounding with explicit cognitive behaviors. On audio understanding and reasoning benchmarks, large audio-language models (LALMs) are limited primarily by perceptual errors, yet existing post-training datasets rely on audio descriptions of limited diversity. Through a four-stage pipeline, LLT (1) builds multi-aspect dense audio descriptions spanning semantic, acoustic, and temporal information, (2) generates more challenging questions, (3) synthesizes incorrect reasoning traces, and (4) extends short reasoning traces into long ones via a thought-continuation strategy that elicits cognitive behaviors such as verification and backtracking. Using LLT, we synthesize 1.6M reasoning traces and train Qwen2.5-Omni-7B with a two-stage pipeline of supervised fine-tuning (SFT) followed by direct preference optimization (DPO). This improves accuracy on MMAU Test-mini (71.5 to 78.2), MMAR (56.7 to 63.7), and MMSU (60.6 to 66.0).