Audio visual models fail global speech vision tasks despite local visual clues
Local Evidence, Global Failure: Diagnosing Speech-Vision Temporal Composition in Audio-Visual Language Models
Computer Vision and Pattern RecognitionArtificial IntelligenceMultimedia
Summary
Sometimes audio-visual language models can find visual clues in video segments that match spoken words, but still fail to decide correctly when looking at the full video. The authors introduce a method called Temporal Evidence Decomposition (TED) that checks local visual evidence separately from how this evidence is combined over time. They show that using TED can greatly improve understanding of where and how these models fail. The work highlights that speech content, timing, and visual evidence need to be treated as separate elements for better video-language model performance.
What this means in practice
- •For video ai developers: Diagnose and improve audio-visual model decision failures by separating local visual verification from speech timing aggregation.
- •For multimedia indexing teams: Enhance video search systems by isolating temporal regions where visual evidence aligns with speech transcripts for more accurate retrieval.
Authors
Jaehong Myung, Seunghun Oh, Unsang Park
Abstract
Audio-visual large language models (AV-LLMs) can fail at full-video speech-vision decisions even when useful visual evidence is locally recoverable. We introduce Temporal Evidence Decomposition (TED), a training-free diagnostic framework that separates local visual verification from speech-relative aggregation. Automatic speech recognition supplies a transcript and utterance interval. TED queries audio-free windows with the transcript for Temporal Alignment and Forecasting, or answer candidates for Cross-Modal Temporal Binding, then aggregates evidence relative to that interval. Across four frozen AV-LLMs on SVHalluc, Temporal Forecasting accuracy rises from 29.3-33.7% under full-video questioning to 79.3-82.2% with TED. Query replacement reduces recovery; the observed anchor remains beneficial when temporal-region availability is held fixed. Physical-isolation and timing-shortcut controls test alternative explanations. The findings distinguish locally recoverable evidence from successful full-video decisions, motivating explicit separation of narration content, utterance timing, and time-indexed visual evidence.