Long video reasoning model improves question answering accuracy
Long-to-Short Video Evidence Reasoning for Grounded Question Answering
Computer Vision and Pattern Recognition
Summary
Answering questions about videos often requires understanding what happens over different time periods. The authors introduce LOVER, a model that learns by first focusing on longer video parts and then shorter ones, improving its ability to find the right video evidence to answer questions. They also designed better ways to reward the model when it spots important video segments and made timestamps easier to see on videos. Their approach works with different underlying systems and sets new performance records on popular video question datasets.
What this means in practice
- •For video content analysis teams: Improve automated systems to find and explain relevant scenes in long videos when answering questions about the content.
- •For video editing software developers: Add features to visualize timestamps more clearly on video frames to assist in precise scene identification during editing.
Authors
Kaiyan Chen, Junbin Xiao, Xun Yang
Abstract
We present LOVER, a \underline{L}ong to sh\underline{O}rt \underline{V}ideo \underline{E}vidence \underline{R}einforced model for grounded question answering (GQA). LOVER highlights three innovations over existing reinforcement-learning (RL) based video reasoning models: (1) \textbf{Long-to-short Video Evidence Curriculum Learning}, which organizes RL training according to evidence duration and progressively adapts the model from long-range grounding to short-term reasoning; (2) \textbf{GQA Rewards}, which underscore the benefit of IoP reward over IoU for evidence spotting rather than strict temporal span overlap; (3) \textbf{Adaptive Timestamp Rendering}, which adaptively renders timestamps onto video frames using background-aware position and color selection to enhance temporal observability. The three designs are model-agnostic and reciprocal. They effectively improve QA, grounding, and grounded QA performance over different backbones. Notably, LOVER built on Time-R1 achieves new state-of-the-art (SOTA) results among open-source models on popular GQA benchmarks: NExT-GQA and ReXTime. Comprehensive ablation studies further validate the effectiveness of our three innovative components.