Totem builds stable 3D object memory for consistent video scenes
Totem: Object-Centric 3D Memory for World-Consistent Video Generation
Computer Vision and Pattern Recognition
Summary
Creating videos where objects look stable and consistent despite big camera movements is very hard. The authors present Totem, a system that remembers each object’s 3D shape and position in a shared space. Using this memory, Totem can generate videos that keep objects looking the same and in the right place even when the viewpoint changes a lot. This approach helps make more realistic and consistent synthetic videos.
What this means in practice
- •For computer graphics developers: Generate consistent 3D scenes in video games or virtual environments with stable object representations during camera moves.$Commercial implications: Enables creating realistic game worlds and virtual scenes that maintain object consistency, enhancing user immersion and visual quality.
- •For augmented reality engineers: Produce world-consistent video outputs to improve AR object placement and tracking under changing viewpoints.
Authors
Zichen Wan, Tianyu Huang, Fan Li, Kai Wang, Hongzhi Zhang, Wangmeng Zuo
Abstract
Generating coherent videos grounded in a unified underlying world remains a major challenge, particularly under large viewpoint changes, where existing methods often struggle to maintain stable representations of individual objects. To this end, we introduce Totem, an object-centric 3D memory framework for world-consistent video generation. Totem first represents each selected object as a persistent 3D memory aligned to a shared world frame, assigning it a dedicated state. Based on these memories, we design a dual-stream 3D conditioning architecture that combines object memories with scene geometry for target-view synthesis. Moreover, these object memories constrain the model's attention with 3D structural information, thereby preserving object surfaces and relative spatial layouts. Extensive experiments on world-generation and object-centric benchmarks demonstrate that Totem generates videos depicting a more object-consistent 3D world under challenging camera trajectories, while supporting versatile applications.