Video agent improves answering multiple questions on long videos efficiently
VAMR: Multi-Question Agentic Reasoning for Efficient Long-Form Video Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding long videos by answering several questions usually means starting over for each question, which wastes time and effort. The authors propose VAMR, a system that handles multiple questions together by exploring the video just once and sharing information across questions. This way, it gathers clues efficiently and builds a shared story to answer all questions better and faster. They trained the system to decide when to answer questions and when to keep looking, achieving better accuracy with fewer steps compared to older methods.
What this means in practice
- •For video content analysis teams: Automate efficient processing of long videos to answer multiple related questions simultaneously, reducing computational cost and time.
- •For surveillance system operators: Improve multi-faceted video review by enabling faster and more accurate answers to various queries about recorded footage.
- •For media production companies: Enhance video archiving and metadata tagging by supplying detailed multi-question answers about long recordings with less effort.$Commercial implications: Supports advanced video indexing and search tools for media firms needing multi-question insights from long content.
Authors
Runquan Gui, Hanzhu Chen, Zehao Wang, Hanxin Zhu, Xin Li, Zhibo Chen
Abstract
Long-form video understanding often involves multiple questions about different aspects of the same recording. Yet existing video agents typically process each question through an isolated tool-use trajectory. This repeatedly restarts video exploration and memory construction, missing opportunities to acquire evidence jointly and progressively build a shared understanding that supports the complete question set. We introduce \textbf{VAMR} (\textbf{V}ideo \textbf{A}gent for \textbf{M}ulti-Question \textbf{R}easoning), which coordinates all questions about a video through one shared tool-use trajectory. At each round, a persistent policy model can invoke tools for one or more unresolved questions and submit answers for questions with sufficient evidence. Question-conditioned visual perception retrieves fine-grained clues for several questions in one call, while layered multi-question memory integrates reusable context into a shared video story and preserves separate evidence for individual questions. After supervised fine-tuning initializes this interaction protocol, we propose question-horizon policy optimization (\qhpo) to optimize shared trajectories in which questions progress and finish at different rounds. Specifically, a question-level critic estimates the value of each active question, while round alignment maps each question advantage to the rounds that directly serve it before the aligned advantages are aggregated to optimize the shared actor. Across LVBench, Video-Holmes, and LongVideoBench, VAMR achieves the highest accuracy overall and the fewest reasoning rounds among iterative methods. On LVBench, it reaches 62.1\% accuracy, exceeding VideoARM by \textbf{4.3} points while reducing reasoning rounds and processed frames by \textbf{85.9\%} and \textbf{61.4\%}.