Video reasoning improves by learning from fewer frames guided by dense views
Less from More: Reinforcing Sparse Video Reasoning from Dense References
Computer Vision and Pattern Recognition
Summary
Many video understanding models think using more video frames always helps with better answers. The authors challenge this idea and focus on making accurate judgments even when only a few frames are available. They created SAVER, a method that learns from detailed videos during training to better understand videos using fewer frames later on. This approach improves performance in several tests and can match models that use many more frames. It means efficient video analysis is possible without always needing to process the full video.
What this means in practice
- •For video content analysis teams: Improve video understanding systems to deliver accurate results using fewer video frames, saving processing time and resources.
- •For autonomous vehicle developers: Enhance real-time decision making by enabling models to reason reliably on sparse video inputs when full frame data is unavailable.
Authors
Wenfang Sun, Yingjun Du, Cees G. M. Snoek
Abstract
Video-language models commonly assume that more temporal observations lead to more reliable reasoning. We question this assumption and argue that the key challenge is not merely processing more video frames efficiently, but learning to reason reliably under limited temporal evidence. We propose SAVER, a dense-to-sparse post-training framework that uses dense video views as training-time references for sparse-frame inference. During reinforcement post-training, paired dense and sparse views are optimized with grounding rewards and a reliability-gated reference reward, encouraging sparse view predictions to preserve task-relevant temporal evidence. Notably, SAVER is trained only on 1,250 randomly sampled temporal grounding examples, without using any video question answering annotations. Across three temporal grounding benchmarks and six video question-answering benchmarks, SAVER consistently improves performance across frame budgets. In particular, SAVER can match or surpass dense-frame Qwen3.5 baselines while using substantially fewer frames. These results show that temporal grounding can serve as an effective evidence-localization proxy for learning sparse video reasoning that transfers to broader video understanding tasks.