Compression and Retrieval: Implicit Memory Retrieval for Video World Models
2026-06-22 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of keeping track of what happens in videos, especially when the camera moves a lot and for a long time. They introduce a new method called Compression and Retrieval (CaR), which uses attention and viewpoint information to smartly remember important parts without heavy computation. They also made a new dataset called SceneFly to help train and test these video memory models. Their experiments show that their method works well and adapts to different kinds of scenes.
video world modelslong-term memoryattention mechanismpositional encodingcontext compressioncamera trajectoriesvideo datasetssynthetic datamemory retrievalbenchmarking
Authors
Zhan Peng, Jie Ma, Huiqiang Sun, Chong Gao, Zhijie Xue, Zhiyu Pan, Zhiguo Cao, Jun Liang, Jing Li
Abstract
Video world models hold promise for simulating interactive environments, yet maintaining consistent long-term memory across complex camera trajectories remains a critical challenge. Existing methods typically rely on computationally expensive context scaling or rigid heuristic retrieval mechanisms, which lacks generalization to varying camera trajectories and environments. In this paper, we propose Compression and Retrieval (CaR), an attention-driven implicit memory retrieval mechanism to overcome these limitations. By injecting viewpoint information via positional encoding, our method performs flexible memory retrieval through attention computation. To efficiently process extended contexts with minimal computational overhead, we further introduce a lightweight context compression network. Furthermore, we construct SceneFly, a large-scale synthetic dataset featuring realistic camera trajectories and frame-level annotations to train and evaluate long-horizon video world models. Extensive experiments demonstrate that our approach achieves state-of-the-art results on established benchmarks and exhibits strong generalization to open-domain scenes.