Audio visual models reduce cross talk errors by focusing on relevant clues
Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation
Artificial Intelligence
Summary
Audio-visual language models sometimes get confused when combining sound and video inputs, leading them to make mistakes called hallucinations. The authors found that trying to use both inputs together can weaken the model’s correct guess when one input alone was enough. They introduced a method called Relevant Evidence Decoding (RED) that figures out which clues (audio, video, or both) matter for each question and gives those clues more weight. This method improved accuracy when tested on multiple benchmarks without needing extra training.
What this means in practice
- •For multimedia app developers: Improve audio and video question answering apps by reducing errors caused by one modality misleading the other.
- •For security monitoring teams: Use enhanced audio-visual models to more reliably identify sounds or objects in surveillance systems by focusing on relevant sensory input.
Authors
Hyunjae Ra, Aecheon Jung, Jungin Park, Sungeun Hong
Abstract
Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions require different perceptual evidence, including audio, video, or their interaction. Notably, we observe that joint audio-visual inference can weaken the prediction even when a model can recover the correct answer from a single informative modality. For example, when asked which instrument is heard, a model may correctly predict violin from the audio alone. Once a video showing a guitar is added, its confidence in violin may drop. In this paper, we introduce Relevant Evidence Decoding (RED), a training-free method that identifies question-relevant evidence and selectively strengthens its contribution. RED uses pointwise mutual information to quantify the predictive support provided by audio and video beyond the question alone. It decomposes their joint contribution into audio, video, and residual interaction components. A question-only inference pass determines the required evidence type, after which the model augments the original audio-visual prediction with the corresponding PMI contribution. Across three audio-visual hallucination benchmarks and three AV-LLMs, RED improves accuracy over standard decoding by up to 7% on CMM, 6.3% on AVHBench, and 3.8% on SVHalluc, with an average relative time to first token of 1.5x standard decoding.