Augmented video models keep scenes and movements consistent longer
GROME: GROunding Factorized MEmory into Video World Models
Computer Vision and Pattern RecognitionMachine Learning
Summary
Keeping videos consistent over long periods is hard when scenes and moving objects change outside the usual short memory window. The authors introduce GROME, a method that separates static parts like backgrounds from moving objects in memory. This helps the video model remember scenes and movements better over time without retraining. It also allows for more flexible video extensions by plugging in new observations without extra training.
What this means in practice
- •For video game developers: Improve in-game scene persistence and character motion over long play sessions using augmented video models that maintain memory without retraining.$Commercial implications: Enables game companies to build more immersive and consistent dynamic environments by extending video memory efficiently.
- •For robotics engineers: Enable robots to maintain coherent environment models over extended operations by separating static scene memory from dynamic object states.
Authors
Fang Yu, Xiaoxue Wu, Haiyu Zhang, Yangfu Li, Botong Zhao, Yu Qiao, Tianfan Xue, Yue Lu, Xinyuan Chen
Abstract
Long-horizon consistency is essential for video world models, yet it becomes difficult to maintain once observed content falls outside the context window. Existing memory mechanisms either retrieve history as implicit context, which remains constrained by context capacity and entangles static and dynamic information or consolidate history into explicit global representations, which struggle to model evolving dynamics and accumulate geometric errors. To overcome these limitations, we introduce \textbf{GROME}, a novel factorized memory framework that augments video world models for long-horizon coherence across both static scenes and dynamic entities. GROME operates via three synergistic components: 1)~\emph{Static-Dynamic Factorization} decouples historical evidence into static and dynamic components, preserving static scene geometry and dynamic entity states separately; 2)~\emph{Local Memory Retrieval} selects spatially aligned static history to eliminate geometric drift alongside the latest dynamic entity states to ensure temporal continuity; 3)~\emph{Training-Free Memory Injection} seamlessly grounds the factorized memory into video world models via their native conditioning pathways. Experiments on long-horizon revisit scenarios demonstrate that GROME substantially enhances scene persistence and dynamic entity fidelity over existing methods, all while preserving the visual quality and controllability of the frozen backbone without additional training. Beyond self-rollouts, GROME readily accommodates external observations as plug-and-play memory, unlocking flexible and controllable video continuation and scene exploration.