Video memory saves time but may miss some answers and costs space
REEL: Same Frames, Different Answers and Different Costs
Computer Vision and Pattern Recognition
Summary
Watching long videos over and over for every question is slow, so systems try to remember parts of the video to answer later questions faster. The authors show that what a system decides to remember affects the answers it can give, and some ways of saving memory lose information needed for some questions. They created REEL, a way to measure and compare different memory strategies using the same video frames, to see how answer accuracy, processing time, and storage size change. They found that some memory methods keep answers almost as accurate as watching the whole video again but need much more storage space. This helps understand trade-offs in video question-answering systems.
What this means in practice
- •For video platform engineers: Optimize video question-answering systems by choosing memory strategies that balance accuracy, processing speed, and storage based on this framework.
- •For cloud service operators: Manage GPU resources and storage costs more effectively for large-scale video analytics using memory trade-off insights from this work.
Authors
Xin Liang, Jiangpeng He
Abstract
Video understanding keeps moving toward longer recordings, and the longer a recording is, the more questions can be asked about it. But processing it again for every question does not scale, so many systems watch the video once, build a memory, and answer later questions from it. Yet what to remember is decided before anyone knows what will be asked, and whatever is left out cannot be recovered later. Moreover, whether the savings pay off is judged by token counts, which fall when the input gets shorter but not when the model reuses what it has already computed. We present REEL, an evaluation framework that compares three saved representations built from exactly the same observed frames, so that a change in answers can come only from what the system remembered. Text writes a short record of what happened, Visual keeps the encoder outputs, and Prefix State also keeps the language-model prefix. Each is compared with full reprocessing (Frames) by paired correctness changes, cumulative GPU time, and storage size. Across six vision-language models and two benchmarks, Text and Frames differ in correctness on 20\% to 29\% of questions, even when their mean accuracy is almost the same. By contrast, Visual and Prefix State stay much closer to Frames. With unlimited storage, Prefix State is the cheapest in total GPU time among memories that keep almost all of the answers Frames gets right, for every model on both benchmarks. The trade-off is storage: Prefix State keeps up to about 21 GiB per video, whereas Text needs only kilobytes. Reusing computation and discarding information both look like efficiency, but only the second removes evidence that later questions may need, and neither mean accuracy nor token count tells them apart. REEL makes that difference measurable for any system that answers many questions from one saved memory.