Video world models learn internal memory retrieval for better scene consistency

Learning to Retrieve: Internalizing Memory Retrieval for Video World Models

Computer Vision and Pattern Recognition

Summary

Generating consistent videos of 3D scenes over time is hard because models forget or drift away from earlier views. This work lets video models learn to remember past scenes by themselves, without separate memory components. The model figures out when to look back and what parts of the past to use based on the camera's position. This helps videos stay realistic and coherent for longer without needing external memory banks.

What this means in practice

  • For virtual reality developers: Create more consistent and immersive 3D scene videos that maintain visual continuity as a user moves through virtual environments.
  • For video game designers: Generate game environments that dynamically retain spatial consistency over long play sessions without external memory systems.

Authors

JiaKui Hu, Tailai Chen, Yuqi Pan, Xuerui Qiu, Jialun Liu, Xiao Cao, Zhenxin Zhu, Guang Chen, Hangjun Ye, Bing Wang, Yanye Lu

Abstract

Video world models aim to generate explorable, 3D-consistent scene videos conditioned on camera trajectories. Existing approaches often rely on external memory systems that explicitly retrieve previously observed content to mitigate scene drift during long-horizon generation. However, these auxiliary memory pathways operate outside the model's internal generative dynamics, preventing the model from intrinsically learning when and what historical information should be retrieved. We propose to internalize memory retrieval into the generation process, allowing retrieval to emerge as an intrinsic behavior of the video world model rather than relying on an external memory system. Based on this principle, we introduce \textbf{Learning-to-Retrieve (L2R)}, which repurposes the model's persistent internal state as a memory for historical context. A camera-conditioned retrieval gate selectively accesses relevant historical information from this state, determining \textit{what to retrieve}, while a retrieval trigger determines \textit{when to retrieve}. We further supervise the trigger with a 3D re-visibility signal, activating retrieval when previously observed content re-enters the current view while otherwise preserving the existing context. Together, these components enable the model to intrinsically acquire memory retrieval behavior and incorporate relevant historical observations into generation without a separate retrieval pathway. Across multiple base models and camera-revisit benchmarks, L2R improves long-term scene consistency while eliminating the need for an external memory bank or 3D conditions. https://jkhu29.github.io/l2r