OmniGround improves timing accuracy in multimodal video and audio analysis
OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training
Computer Vision and Pattern Recognition
Summary
It is hard for large language models to accurately find the exact times in videos or audio where events happen. The authors found that the model's internal focus actually points to the right moments better than its final answers. They created a new training method that aligns what the model focuses on with its predictions and the correct times. They also built a large training set to help the model learn better. Their improved model, OmniGround, is better at pinpointing when things happen in videos and audio without needing extra timing clues.
What this means in practice
- •For video streaming platforms: Automatically identify precise moments in videos matching text or audio queries for improved content search and navigation.$Commercial implications: Enables precise video timestamp retrieval for enhanced search features that streaming platforms can offer to users.
- •For media content indexing teams: Create accurate temporal annotations for videos and audio recordings to organize large multimedia archives efficiently.
Authors
Zixuan Chen, Depeng Wang, Hao Lin, Ke Xu, Ya Guo, Huijia Zhu, Tanfeng Sun, Xinghao Jiang
Abstract
Omni-modal large language models can jointly understand video, audio, and text, yet they remain weak at temporal grounding, a capability essential for real-world audio-visual applications. To understand this limitation, we look beyond final predictions and examine the model's internal temporal evidence. We identify a temporal generation gap: the internal attention of the model localizes the target moment more accurately than the timestamps it generates. To close this gap, we propose TACG (Temporal Attention-Consistency Grounding), a post-training method that treats internal attention as a reliable temporal pivot and aligns it with both the model prediction and the ground-truth annotation through attention-consistency training. We further build a 580K-sample training dataset by reconstructing temporal QA from dense audio-visual annotations and organizing general omni-modal instruction data. Using TACG and this training dataset, we train OmniGround, a general-purpose omni-modal model with stronger temporal grounding capability, without injecting any additional temporal cues. OmniGround reaches state-of-the-art omni-modal temporal grounding, including 37.5 mIoU on LongVALE and 35.7 mIoU on UnAV-100, while staying competitive on both general audio-visual understanding and vision-only grounding.