Papers for

media content indexing teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

OmniGround improves timing accuracy in multimodal video and audio analysis

OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training

Abstract: Omni-modal large language models can jointly understand video, audio, and text, yet they remain weak at temporal grounding, a capability essential for real-world audio-visual applications. To understand this limitation, we look beyond final predictions and examine the model's internal temporal evidence. We identify a temporal generation gap: the internal attention of the model localizes the target moment more accurately than the timestamps it generates. To close this gap, we propose TACG (Temporal Attention-Consistency Grounding), a post-training method that treats internal attention as a reliable temporal pivot and aligns it with both the model prediction and the ground-truth annotation through attention-consistency training. We further build a 580K-sample training dataset by reconstructing temporal QA from dense audio-visual annotations and organizing general omni-modal instruction data. Using TACG and this training dataset, we train OmniGround, a general-purpose omni-modal model with stronger temporal grounding capability, without injecting any additional temporal cues. OmniGround reaches state-of-the-art omni-modal temporal grounding, including 37.5 mIoU on LongVALE and 35.7 mIoU on UnAV-100, while staying competitive on both general audio-visual understanding and vision-only grounding.

Mon 19 OctComputer Vision and Pattern Recognition
The gist
It is hard for large language models to accurately find the exact times in videos or audio where events happen. The authors found that the model's internal focus actually points to the right moments better than its final answers. They created a new training method that aligns what the model focuses on with its predictions and the correct times. They also built a large training set to help the model learn better. Their improved model, OmniGround, is better at pinpointing when things happen in videos and audio without needing extra timing clues.
Open → 2610.25929v1