Large audio language models reduce false yes answers with token adaptive decoding
TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models
SoundComputation and LanguageComputer Vision and Pattern Recognition
Summary
Large audio-language models sometimes wrongly say 'yes' to sounds that aren't actually present, which can cause mistakes in audio question answering. The authors propose a way called Token-Adaptive Decoding (TAD) that helps the model decide more accurately by comparing what it hears against silence. This method adjusts decisions based on confidence and avoids unnecessary corrections. Tests show that TAD improves accuracy compared to earlier methods, especially in challenging cases.
What this means in practice
- •For audio system developers: Improve reliability of audio question answering systems by reducing false positive detections of absent sounds using confidence-based token decoding.
- •For voice assistant engineers: Enhance voice assistants’ understanding of audio events by incorporating contrastive decoding strategies that prevent hallucinated sound recognition.
Authors
Heyu Chang, Nianwen Si, Hao Zhang, Wenlin Zhang, Dan Qu
Abstract
Large audio-language models (LALMs) can hallucinate audio objects, answering "yes" to absent sound events, thus undermining reliability in audio question answering. We propose Token-Adaptive Decoding (TAD), a training-free strategy for hallucination mitigation that grounds the initial yes/no decision by contrasting logits under real audio with a matched silent reference. TAD introduces a token-adaptive, confidence-guided gate that is decision-critical at the first decoding step and class-conditional on affirmative tokens, using the audio-silent margin to avoid overcorrection when evidence is weak or already sufficient. Experiments on AudioCaps-Hallucination show that, relative to Audio-Aware Decoding (AAD), a contrastive baseline with fixed contrast strength, TAD improves F1 for Qwen2 by 0.059 to 0.117 across Popular, Adversarial, and Random splits, and for Gemma by 0.025 to 0.064, while on Clotho-AQA it raises F1 from 0.810 to 0.816 on Qwen2 and remains comparable to AAD on Gemma.