VideoEvolve improves long video understanding by evolving memory and retrieval
VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding long videos is hard because you need to remember many details and find the right ones when asked questions. The authors show that just changing how you look up memories isn’t enough if what’s saved is fixed and possibly missing important info. They created VideoEvolve, a system that improves itself by gradually adjusting both what it remembers and how it retrieves that information. This back-and-forth learning helps the system get better at answering complex questions about long videos.
What this means in practice
- •For video analytics teams: Enhance automated analysis of extended video footage by jointly improving stored video features and retrieval methods for more accurate insights.
- •For surveillance system engineers: Improve long-duration video monitoring by adaptively evolving memory and retrieval to better identify relevant events and reduce missed details.
Authors
Yongchao Xu, Bowen Ye, Jiefeng Gan, Junkai Ma, Wenzhao Li, Sen Tao, Yi Wei, Jiawei Liu
Abstract
Long video understanding increasingly relies on external memory to organize massive visual streams into compact representations. However, most memory-based methods dynamically adapt how information is retrieved for different questions, while largely fixing what is remembered. This mismatch makes missing details costly to recover, whereas stored information is valuable only when it can be reliably retrieved. To address this issue, we propose VideoEvolve, a novel self-evolving framework that jointly evolves memory and retrieval for long video understanding. Specifically, starting from a coarse low-frame-rate overview, VideoEvolve couples a Memory Evolver for selective memory augmentation with a Retrieval Evolver for adaptive retrieval over the evolving memory. We then co-evolve the two Evolvers through alternating agentic reinforcement learning (Agentic RL), updating one while freezing the other. To steer this alternating evolution, Bottleneck-Aware Evolution Feedback (BEF) identifies whether the current bottleneck lies in memory or retrieval and directs optimization toward the more limiting side. Furthermore, VideoEvolve introduces Capability-Aware Evolution Feedback (CEF) to alleviate downstream feedback from over-specializing memory to a fixed set of training questions, shifting training toward underdeveloped yet learnable video capabilities. By integrating Agentic RL with BEF and CEF, VideoEvolve transforms downstream reasoning experience into transferable capability updates, providing a concrete path from static long-video systems toward experience-driven, self-improving multimodal intelligence. Extensive experiments on multiple long video understanding benchmarks demonstrate the effectiveness of VideoEvolve.