MemEvo automatically finds efficient memory methods for streaming video
MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms
Artificial Intelligence
Summary
Understanding videos that keep streaming endlessly is hard because the computer needs to remember important parts without knowing what questions will come later. The authors came up with a new way to automatically discover how to best store and summarize video information using a special program language and a powerful language model. Their system tests many ways to handle memory and learns which ones work best, without changing the main video language model. This method helps computers understand streaming video better while using memory and computing power wisely.
What this means in practice
- •For video platform engineers: Improve real-time video analysis systems by automatically finding memory use strategies that keep key information without overload.
- •For security monitoring teams: Enhance continuous video surveillance by efficiently compressing live streams into manageable memory while maintaining important visual details for queries.
Authors
Guohong Liu, Jialei Ye, Shanhui Zhao, Yunxin Liu, Yuanchun Li
Abstract
Query-agnostic streaming video understanding requires vision-language models to continuously compress an indefinitely growing visual stream into a bounded memory before future queries are known. The performance depends critically on the memory mechanism--what observations to preserve, how to represent and consolidate them, and what information to retrieve when a query eventually arrives. Rather than designing a single memory architecture by hand, we formulate memory design as a search problem over executable memory programs. We introduce a lightweight domain-specific language that expresses memory mechanisms through structured primitives for representation, admission, retention, consolidation, budgeting, and retrieval, while enforcing causal and bounded-memory constraints. Although structured, the derived program space remains large and contains heterogeneous, conditionally dependent design choices whose effects can only be assessed via downstream execution. We therefore propose MemEvo, an LLM-driven auto-research framework that uses pretrained LLM as a semantics-aware proposal model to iteratively generate and refine candidate memory programs based on accumulated experimental feedback. At runtime, a deterministic evaluation pipeline validates and evaluates each candidate, while the underlying vision-language model remains frozen throughout discovery. We finally produce a training-free, bounded-memory mechanism. Extensive experiments on StreamingBench and OVO-Bench demonstrate strong streaming video understanding performance together with substantial context and inference efficiency.