Video agents learn to reason and decide where to look next
Joint Latent Reasoning and Observation for Agentic Video Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding videos is hard because a smart system needs to both answer questions and figure out where to look next in the video for more clues. The authors created a method called LARVA that teaches smaller video agents to think and observe at the same time using hidden signals, not just words. This helps the agents decide what to watch next and improve their answers by learning from bigger, smarter systems. Their tests show this approach works better on different video tasks compared to methods only relying on text explanations.
What this means in practice
- •For autonomous vehicle engineers: Improve in-car video systems to actively search for relevant information when interpreting road scenes.
- •For security monitoring teams: Develop video surveillance systems that dynamically focus on new important areas to detect and explain events.
Authors
Junzhang Liu, Hao Yuan, Qi Zhang, Yiming Wu, Long Chen, Zhecan Wang
Abstract
Effective video agents must learn not only how to answer, but also where to look next. Transferring this capability to smaller models is challenging because each observation serves two purposes: supporting an answer and guiding the search for further evidence. Textual descriptions may capture what is currently visible while omitting cues needed for subsequent acquisition. We introduce Latent Agentic Reasoning for Video Analysis (LARVA), a framework that distills multi-round video agents through jointly learned latent reasoning and visual observations. Rather than communicating through captions, an observer passes compact continuous representations to a reasoner that selects the next acquisition action or produces the final answer. Crucially, action and answer supervision propagate through this interface, training the observer to communicate information useful for downstream decisions. Observation reconstruction and reasoning-distribution alignment provide auxiliary supervision from textual teacher trajectories. Across five video-understanding benchmarks, LARVA achieves strong performance with 4B and 7B backbones. On VideoHolmes, Video-MME, and LVBench, the 7B model outperforms a textual CoT student trained on the same teacher trajectories by 5.1, 2.5, and 3.7 percentage points, respectively. These results highlight the value of jointly learning how agents represent evidence and act on it when distilling video-understanding capabilities.