Spatio temporal memory improves long video generation consistency
MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation
Computer Vision and Pattern RecognitionGraphics
Summary
Generating videos that last a long time is hard because computers only remember a limited amount of what happened before, causing details to get lost. The authors present MosaiChunk, a way to store important visual memories at different times and places in the video and pick the best parts to help the generator recreate scenes accurately even after a long time. They train a small system to select which memories to use while keeping the video generator itself unchanged. Their tests show MosaiChunk helps video generation remember details better than previous methods, making revisits in videos more consistent over time.
What this means in practice
- •For video production teams: Enhance video editing tools to maintain visual details in long generated video sequences by improved memory retrieval techniques.
- •For game developers: Create more consistent and detailed long animations in games by integrating spatio-temporal memory for recollecting past scenes.
Authors
Yiwen Zhang, Haocheng Xi, Michael Tian-Yue Liu, Alexei A. Efros, Hadar Averbuch-Elor, Qianqian Wang, Haiwen Feng
Abstract
Long-horizon autoregressive video generation is limited by a finite context window. When an object or scene falls out of context, its fine-grained visual details may be lost and difficult to recover upon reappearance. To retain access to such visual details, we introduce MosaiChunk, a spatio-temporal memory mechanism that composes a mosaic of selected historical key-value (KV) entries across space and time. Our approach is motivated by the observation that a frozen video generator can directly consume such non-contiguous historical KV and recover the corresponding visual content. We therefore keep the generator fixed and learn only a lightweight router that determines which historical sections to include in the mosaic under a fixed active-memory budget. We further introduce RememBench, a benchmark of long-horizon revisits with prompt-driven text-to-video (T2V) and camera-driven image-to-video (I2V) splits. Our experiments show that MosaiChunk consistently improves revisit consistency over both sliding-window inference and whole-chunk retrieval under matched memory budgets, across both T2V and I2V settings.