Audio language models improved by reducing hallucinations through preference learning
Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization
Sound
Summary
Large models that understand sound and text together sometimes imagine things that aren't there, especially when mixing audio with written words. The authors found a way to train these models better by making them compare normal sounds with altered versions, so they focus more on the actual audio. This method helped improve the model's ability to understand when sounds happen and whether certain sounds exist in a clip. They showed this improvement on two different audio analysis tasks with big accuracy gains.
What this means in practice
- •For audio software engineers: Develop audio analysis tools that more accurately detect sound events with fewer mistakes caused by mixing text and audio inputs.
- •For speech recognition developers: Improve speech recognition systems by ensuring models rely more on actual audio features instead of misleading textual context.
Authors
Bebe Cosgrove, Aaron Isidore Grace, Weiran Wang
Abstract
Large Audio Language Models (LALMs) are prone to hallucinating and over-relying on text priors when simultaneously presented with audio and text inputs. To mitigate these hallucinations, we propose utilizing the multimodal Direct Preference Optimization (mDPO) objective, which forces the model to ground its generation in the acoustic input by contrasting intact and distorted audio counterparts. We extend this preference learning framework to the audio domain by applying a variety of acoustic perturbations. Evaluating the Qwen2-Audio backbone across the DCASE 2025 Challenge and AH Existence datasets, we demonstrate that extending mDPO to LALMs significantly enhances temporal reasoning in the complex DCASE dataset, and improves performance on basic existence verification in the AH benchmark. We identify temporal reversal, frequency masking, and random noise as the most effective perturbations. Ultimately, our approach achieves an absolute accuracy improvement of 14.0% on the DCASE 2025 dataset and 27.4% on AH Existence.