Incentivizing Vision Language Models to Search for Long Video Question Answering
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors present VSeek, a system that improves understanding of long videos by turning the task into a back-and-forth search rather than trying to answer questions in one go. They use natural language queries to find important parts of the video and train the system with reinforcement learning using a new method that checks if specific visual elements mentioned in the question actually appear. This approach helps the system get better at finding useful clips, leading to higher accuracy in video question answering tests. The authors also provide their code for others to use.
Long-video question answeringAgentic frameworkReinforcement learningNatural language searchTemporal logicVisual verificationNeuro-symbolic approachMulti-turn retrievalVideo understandingPass@1 / Pass@4
Authors
Harsh Goel, S P Sharan, Sahil Shah, Minkyu Choi, Joungbin An, Kristen Grauman, Sandeep P. Chinchali
Abstract
We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process. VSeek utilizes a natural language-driven search to identify relevant context within long videos and is post-trained with reinforcement learning (RL) to jointly formulate targeted search queries and reason over retrieved clips for LVQA. While RL post-training has revolutionized reasoning in symbolic domains such as mathematics and code, its application to long-video understanding remains hindered by a lack of verified rewards. To ensure that the retrieved context is relevant, we propose a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification. Specifically, complex user queries are compiled into formal temporal logic specifications for systematically decomposing natural language questions into a definitive checklist of required atomic visual primitives, such as key objects and activities, along with their temporal ordering. These systematically derived grounding events provide the critical feedback signal for RL post-training, enabling dense, verifiable rewards based on the successful retrieval of these specific visual elements rather than relying entirely on outcome-only answer accuracy. By explicitly optimizing for this verifiable evidence-seeking behavior, VSeek improves Pass@1 scores by up to 8% and Pass@4 scores by 15% on long-video understanding benchmarks compared to base models. We open-source our code at https://utaustin-swarmlab.github.io/VSeek.