Video agent learns to explore and understand long videos efficiently
VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding long videos is hard because you can only look at a few frames at a time. The authors treat this problem like an agent reading a book chapter by chapter—deciding how fast to read, which parts to remember, and when to go back for unclear sections. They built VideoScout, a smart system that adapts its watching speed to balance thoroughness and efficiency. It learns from lots of examples and improves through trial and error to better answer questions about long videos.
What this means in practice
- •For video analysis teams: Develop tools that automatically interpret and answer questions about hours-long video content by focusing on key moments dynamically.
- •For surveillance monitoring staff: Create systems that quickly identify important evidence in long surveillance footage by adjusting viewing speed and revisits intelligently.
Authors
Weixin Xu, Zhenyu Yang, Bing Wang, Shengsheng Qian, Changsheng Xu
Abstract
Multimodal Large Language Models (MLLMs) have achieved remarkable progress on short video understanding yet remain limited on long videos due to the limited visual context window. Prevailing approaches rely on uniform frame sampling or recent coarse-to-fine agentic zooming, both of which struggle to localize sparse, decisive evidence in sufficiently long videos. We formulate long video understanding as a \textbf{Sequential Evidence Acquisition (SEA)} problem, in which an agent reads the video turn by turn along the temporal axis, deciding at each turn how fast to watch, what evidence to retain, when to revisit uncertain segments, and when to stop and answer. Inspired by this view, we propose \textbf{VideoScout}, a multi-turn reasoning agent that instantiates the SEA paradigm through adaptive reasoning pacing. Specifically, by dynamically controlling the viewing pace, VideoScout enables efficient traversal of long videos within a bounded visual context window, allowing the agent to access more video content while balancing content analysis depth with reading efficiency. To train VideoScout, we construct VideoScout-66K, a set of over 66K high-quality exploration turns from 10K answer-verified trajectories, and adopt a two-stage pipeline: cold-start supervised fine-tuning teaches the agent per-turn output format, while the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm performs trajectory-level reinforcement learning with a composite reward that jointly considers answer accuracy, output format compliance, and the temporal alignment between the agent's viewing progress and the teacher's answer timing measured by intersection-over-union (IoU). Extensive experiments on long video understanding and reasoning benchmarks demonstrate that our 7B model achieves strong performance compared with existing trained 7B agentic models.