Large audio language models reduce false sounds with adaptive decoding
TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models
SoundComputation and LanguageComputer Vision and Pattern Recognition
Summary
Large audio-language models sometimes mistakenly say that sounds are present when they are not, which can cause incorrect answers. To fix this, the authors developed a method called Token-Adaptive Decoding (TAD), which compares the model’s confidence in hearing a sound with its confidence when there is actually silence. This approach adjusts its decisions based on how sure it is and avoids changing answers too much when it already has enough evidence. The authors tested TAD on different datasets and found that it helped reduce false positive sound detections better than previous methods.
audio-language modelshallucinationtoken-adaptive decodingcontrastive decodinglogitsconfidence gatingaudio question answeringF1 scoresilent referencehallucination mitigation
Authors
Heyu Chang, Nianwen Si, Hao Zhang, Wenlin Zhang, Dan Qu
Abstract
Large audio-language models (LALMs) can hallucinate audio objects, answering "yes" to absent sound events, thus undermining reliability in audio question answering. We propose Token-Adaptive Decoding (TAD), a training-free strategy for hallucination mitigation that grounds the initial yes/no decision by contrasting logits under real audio with a matched silent reference. TAD introduces a token-adaptive, confidence-guided gate that is decision-critical at the first decoding step and class-conditional on affirmative tokens, using the audio-silent margin to avoid overcorrection when evidence is weak or already sufficient. Experiments on AudioCaps-Hallucination show that, relative to Audio-Aware Decoding (AAD), a contrastive baseline with fixed contrast strength, TAD improves F1 for Qwen2 by 0.059 to 0.117 across Popular, Adversarial, and Random splits, and for Gemma by 0.025 to 0.064, while on Clotho-AQA it raises F1 from 0.810 to 0.816 on Qwen2 and remains comparable to AAD on Gemma.