Self supervised method improves time accuracy in video language models
SAPO: Self-Supervised Attention Policy Optimization for Improving Temporal Video Grounding in Vision-Language Models
Computer Vision and Pattern Recognition
Summary
It can be hard for computers to tell exactly when something happens in a video just by reading a description. The authors found that current video language models already know roughly the right time but struggle to give exact timestamps. They created a new way called SAPO that uses the model’s own internal attention clues to teach itself better timing, without needing extra labeled examples. This method improves the model’s ability to pinpoint when events occur in videos, and it works across different models and datasets.
What this means in practice
- •For video content indexing teams: Improve accuracy of video search by refining event time predictions without needing expensive labeled data.
- •For multimedia platform engineers: Enhance video clip recommendations by better matching textual queries to precise video intervals using self-supervised methods.
Authors
Zixuan Chen, Depeng Wang, Wei Chen, Ke Xu, Ya Guo, Huijia Zhu, Tanfeng Sun, Xinghao Jiang
Abstract
Video vision-language models (VLMs) have strong semantic understanding, yet remain unreliable at precise temporal grounding. Existing gains from explicitly aligning video frames with absolute time suggest that VLMs may already encode accurate temporal locations internally but lose precision when expressing them as timestamps. Our analysis across four VLMs and two grounding datasets confirms this hypothesis: attention intervals outperform predicted intervals on approximately 66% of valid examples. We term this discrepancy the temporal generation gap. Because precise temporal annotations are expensive, this finding motivates closing the gap to improve temporal grounding using only unlabeled video-query pairs. We propose Self-Supervised Attention Policy Optimization (SAPO), which uses internal temporal attention as self-supervision for Group Relative Policy Optimization and applies a ground-truth-free criterion to select high-value samples. SAPO consistently improves temporal grounding across multiple datasets and models spanning weak to strong initial grounding ability. Relative mIoU gains reach 33.7% on Qwen2.5-VL-3B and 5.4% on the strong Time-R1-7B. The improvements transfer across datasets while preserving general video understanding.