Large audio language models show model-specific sound hallucination patterns
Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination
Artificial IntelligenceMachine LearningSound
Summary
Some advanced audio-language computer models sometimes imagine sounds that aren’t really there, called hallucinations. The authors tested a new way to find out if these imagined sounds come from specific parts of the audio or just random places. They found that the location in the audio does matter, but it depends on the model. Their method also improved the model’s accuracy better than previous tricks. This helps understand and reduce errors when machines listen and talk about sounds.
What this means in practice
- •For audio software engineers: Improve model decoding by combining localized audio evidence removal to reduce hallucinations and increase transcription accuracy.
- •For speech recognition teams: Use causal evidence localization methods to better detect and mitigate incorrect sound claims made by audio-language models.
Authors
Mohammad Khalooei, Mohammad Sabokrou
Abstract
Large audio-language models (LALMs) frequently exhibit object hallucination, affirming absent sounds. The dominant mitigation, Audio-Aware Decoding (AAD), contrasts logits under real audio against an all-zero "blank" waveform, flagged by prior review as out-of-distribution. We introduce a black-box causal test that isolates the claimed evidence itself: localize a claim's evidence region via CLAP similarity, remove it by spectral gating, and compare against a random-window control across a 2x2x2 grid on three LALMs. On Qwen2-Audio-7B, localized ablation's raw AUROC trails blank's (0.613 vs. 0.721); this is a duration confound, not a localization failure, since blank removes the entire clip versus the grid's 2-4 s. Holding duration fixed, localization beats random removal at every cell on both models. Raw detection value remains model-dependent, especially on Audio Flamingo 3. The same evidence corrects decoding: contrastive decoding combining blank and language-prior contrasts, gated and ensembled, raises Qwen2-Audio accuracy from 57.1% to 71.9%, beating AAD alone. A unified causal-grounding profile does not transfer between models, indicating different grounding regimes. The source code and a detailed report of our findings, including identified limitations, are available at https://github.com/khalooei/causa.