Video memory system improves question answering on long videos
Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos
Computer Vision and Pattern RecognitionMachine Learning
Summary
Long videos are hard for current AI systems to understand because they are too big to look at all at once. The paper introduces Video-EMem, a special way to create a compact memory from a video that keeps important visual and text details and can be reused to answer many questions without reprocessing the whole video. The system stores the video’s information efficiently and improves accuracy and speed when asking questions about the video. The authors show their method works better than just feeding parts of the video directly to AI models.
What this means in practice
- •For video analytics teams: Build tools that answer many queries about long videos efficiently by storing compact reusable video memories instead of reprocessing the full footage each time.
- •For media content platforms: Improve user search experience on long-form videos by enabling faster and more accurate question answering using a compact video memory representation.$Commercial implications: This enables premium video search products that offer quick, accurate answers over large video libraries, differentiating content platforms.
Authors
Kerui Chen, Jinglu Wang, Yun Wang, Xinjiang Wang, Yan Lu
Abstract
Long videos often exceed the context capacity of multimodal large language models (MLLMs), while agentic search repeats costly video processing and grounding for each query. Reusable memory can amortize this cost, but must preserve information for questions unknown during construction. We propose Video-EMem, an evolvable video memory harness built on time-anchored perceptual observations. This representation combines compact free-form text with sparse images to preserve semantic content and complementary visual details across temporal scales, supporting temporal navigation without a fixed semantic schema. A Writer encodes each video once by consolidating overlapping multiscale observations, and a Reader retrieves and organizes evidence solely from the resulting memory. With the underlying models frozen, we evolve their workflows and operators using distinct feedback: query-independent source fidelity and coverage for the Writer, and QA failures in evidence localization and utilization for the Reader. Experiments support this separation for generalization beyond development queries, with our evolution strategy improving average accuracy by 7.2% over the unevolved harness. While reducing storage by about 99.86% on average relative to raw videos, Video-EMem improves Qwen3-VL-30B by an average of 2.6% across three long-video QA benchmarks over direct input of 2,048 frames. Using the same o3 answer model as the state-of-the-art agentic method, it improves average accuracy by 2.5% with lower per-query latency.