Slot memory enables long term video generation from changing views
Long-horizon Video Generation with Slot Memory Addressed By Region Tokens
Computer Vision and Pattern Recognition
Summary
Generating videos that show scenes over long time periods is hard because computers must remember what the scene looked like before, without using too much memory. The authors introduce SMART, a new method that stores visual information in fixed-size memory slots and retrieves it based on where the camera is looking, not how recently the scene was seen. This approach lets the system recall details accurately even after long times, without needing to build a full 3D model or keep growing memory. SMART also estimates how confident it is about recalled details to improve video generation quality.
What this means in practice
- •For game developers: Generate consistent, high-quality video sequences of game scenes with limited memory, even when the camera revisits places after long intervals.$Commercial implications: Enables game engines to create immersive, realistic environment videos efficiently for extended gameplay and streaming.
- •For video production teams: Create long videos of scenes where the camera moves through complex environments, maintaining detail without costly 3D reconstruction or memory.
Authors
Eren Cetin, Julius Erbach, Vukasin Bozic, Hyojun Go, Zixiang Zhao, Nikolai Kalischek, Federico Tombari, Konrad Schindler, Dominik Narnhofer
Abstract
Memory for long-horizon video generation must satisfy two competing requirements: it must remain bounded so generation can continue indefinitely, while still recovering what was observed at a place when the camera returns, regardless of how long ago it was seen. Reconstruction-based memories provide spatial recall by caching and reprojecting explicit geometry, but require growing storage and depend on estimated 3D structure. Recurrent memories keep a constant-size state, but retention is governed by temporal recurrence, causing old observations to fade with time. We introduce SMART, a slot memory addressed by region tokens, which combines bounded storage with spatially addressed recall. SMART stores content in a constant-size slot recurrence whose writes are addressed by the observing camera and whose reads are queried by the rendering camera, without constructing an explicit 3D representation or maintaining a growing cache. Consequently, recall is conditioned on inter-view geometry rather than on recency. Each read additionally provides a confidence estimate that conditions a chunk-causal diffusion transformer, allowing it to selectively trust retrieved content while relying on its generative prior elsewhere. On long trajectories in Memory Maze and Minecraft, SMART maintains high-fidelity recall far beyond its training horizon while using constant memory and per-step cost.