Video models improve reasoning by revisiting visual evidence efficiently
VideoLVR: Latent Visual Reasoning over Grounded Video Evidence
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding videos can be hard for AI because they often forget important visual details while thinking. The authors present VideoLVR, a method that helps AI keep track of key visual parts of a video during its reasoning steps. Instead of remembering video details by text or huge clips, their system encodes important visuals into compact tokens it can revisit easily. This process helps the AI answer questions about videos more accurately without needing to load too much data at once.
What this means in practice
- •For video analytics developers: Enable video understanding systems to track and reason about visual evidence efficiently without large memory increase, improving task accuracy.
- •For multimodal ai system builders: Incorporate latent visual tokens as part of reasoning actions to enhance video comprehension models under reinforcement learning.
Authors
Shilong Li, Xingyuan Bu, Hao Shi, Yun Xing, Weixin Feng, Qiangpeng Yang, Shilei Wen
Abstract
Chain-of-thought reasoning has substantially advanced video understanding in multimodal large language models (MLLMs). However, language-based reasoning traces do not provide a reliable mechanism for models to revisit relevant visual evidence, leading to visual forgetting and impairing their ability to reason over video content. Existing approaches maintain visual grounding during reasoning in two main ways: they either refer to evidence using textual coordinates, which specify symbolic locations rather than the visual content itself, or reinsert cropped regions or frames into the multimodal context, preserving visual details at the cost of substantial context expansion. To address these limitations, we introduce VideoLVR, which constructs latent visual evidence from local regions at relevant video moments, thereby providing visual support for intermediate reasoning steps. To supervise latent visual evidence generation, we introduce a visual token prediction objective that formulates visual token reconstruction as prediction over a vocabulary derived from the input video, thereby unifying visual evidence reconstruction and text generation within the same token prediction framework. This shared prediction framework enables MLLMs to reason over interleaved text tokens and latent visual states while integrating evidence from different moments. We further extend VideoLVR to reinforcement learning by treating sampled visual tokens as actions alongside text tokens and jointly optimizing the policy using final-answer rewards. By efficiently revisiting task-relevant visual information during reasoning, VideoLVR improves video understanding without substantially expanding the multimodal context. Experiments on multiple video benchmarks demonstrate its effectiveness, especially on tasks requiring fine-grained visual evidence.