Audio visual models struggle to judge timing and event order in videos
AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding what happens in a video means not just describing scenes but also knowing when events start, how long they last, and if sound and picture match up. The paper presents AVTrace, a set of tests designed to see how well advanced video-audio AI models can handle these timing challenges. The authors found that current top models often fail on tasks like syncing sound and video and following event sequences well. They also show that training models specifically on timing can improve their performance on some of these tasks.
What this means in practice
- •For video platform developers: Improve video search and content indexing by testing models on AVTrace to better locate events in time and synchronize audio with visuals.
- •For audio equipment designers: Use AVTrace evaluation to enhance devices that rely on audio-visual sync detection and correction for better user experience.
Authors
Longyin Zhang, Parth Sakhare Mahendra, Chengwei Wei, Ning Zhang, Lim Ming Chong, Sirui He, Ai Ti Aw
Abstract
Omni models can describe video content, but can they locate events in time, preserve event order, and judge audio-visual synchronization? We introduce AVTrace (Audio-Visual Temporal Reasoning Assessment and Capability Evaluation), a silver-standard diagnostic suite spanning onset and span grounding, synchronization, next-step prediction, cross-modal localization, chain parsing, and event-conditioned comprehension. It contains 34,114 training examples and category-balanced development and test splits of 3,500 and 7,000 examples. We evaluate five open omni models under their respective input configurations using reference-blind response normalization followed by deterministic scoring. All five off-the-shelf systems score below the test split's majority-label baseline of 0.556 on synchronization verification, and obtain low scores on chain parsing and event-conditioned grounding and comprehension. Development-set perturbations reveal task-dependent sensitivity in Qwen3-Omni-30B to modality removal and changes in visual input processing, without isolating their underlying causes. Parameter-efficient temporal post-training improves Gemma4-E4B-it on several benchmark metrics. On three external image benchmarks, task metrics change modestly, including some degradations, while teacher-forcing perplexity decreases. Together, these findings show that semantic reference-text overlap should not be treated as a proxy for temporal localization, and that AVTrace can identify task-specific weaknesses while providing a testbed for temporal post-training.