Video evidence indexing improves question answering on long videos under token limits
Video Evidence Indexing: Learning Where to Look from Video Previews for Token-Budgeted Long-Video Question Answering
Computer Vision and Pattern Recognition
Summary
Answering questions about long videos is hard because computers can only look at a limited number of high-quality video parts at once. The authors propose a way to first look quickly at a low-quality preview of the whole video to guess which parts might have important information. Then, the model spends its limited attention on those important moments at higher quality to answer the question. They train the system so it learns when to look closely, even without exact guidance about which frames are important. This approach helps the system answer questions more accurately using fewer video tokens.
What this means in practice
- •For video analytics developers: Build video question answering tools that target important moments in long videos using fewer high-resolution frames for efficiency.
- •For surveillance system engineers: Improve event detection by focusing computational resources on key video segments identified from low-resolution previews.
Authors
Haowen Guan, Shengzhi Li, Shichao Pei
Abstract
Long-video question answering is limited by the high cost of visual tokens and by the fixed context width of current VLMs. A long-video question may require broad temporal coverage, but the answer is often supported by only a compact set of moments. To locate these moments efficiently, we propose token-budgeted Video Evidence Indexing (VEI): given a dense low-resolution Video Preview, the model constructs a compact high-resolution Evidence Set for final reasoning. We treat VEI as a policy that must jointly solve \textit{evidence localization}, which finds question-relevant moments, and \textit{budget planning}, which decides where to spend the limited high-resolution frame budget. We implement this idea with an inference pipeline: the Video Preview provides cheap global coverage, Video Evidence Indexing constructs the Evidence Set, and Answer Generation combines both inputs for final VQA. To address missing frame-level supervision, we adopt privileged self-distillation, where an answer-aware teacher guides the normal test-time policy on student-generated indexing traces. We explore previews at 1, 6, 12, and 24 visual tokens per frame, training a single policy that supports all four resolutions. Experiments show that Video Evidence Indexing improves accuracy under limited visual budgets, and self-distillation further improves both QA accuracy and temporal evidence localization.