Video world models improve memory to reduce visual errors
GLIMPSE: Global-Local Indexing with Multi-Precision Spectral Encoding for Explorable Video World Models
Computer Vision and Pattern Recognition
Summary
When a camera moves around a scene and then looks back, small errors in understanding the scene’s depth or position can cause visual glitches like ghost images or duplicates. The authors present GLIMPSE, a method that uses a combination of local and global memory to keep track of the scene more accurately. This approach adjusts how precise the location information is, depending on uncertainty, reducing mistakes when projecting parts of the scene. Their method leads to better consistency in how the scene looks from different viewpoints.
What this means in practice
- •For augmented reality developers: Improve scene continuity and reduce ghosting when rendering objects from changing camera views in AR environments.$Commercial implications: Enables more visually consistent AR experiences by reducing rendering errors during user motion.
- •For robotics teams: Enhance the accuracy of visual memory for robots navigating dynamic environments from camera input.
Authors
Zhufeng Xu, Henglei Lv, Bailin Deng, Yikang Ding, Xiaoqiang Liu, Guoxin Zhang, Hongbo Fu, Lin Gao
Abstract
A camera-controlled video world model must keep a scene consistent when the camera returns to it. Memories that render stored features into the target frame can turn depth and pose errors into ghosting or duplicated structure. Positional transport avoids that resampling by relocating a historical patch's address rather than its content, but assigns full positional precision even to inaccurate projections. We present GLIMPSE, a global-local memory interface with band-wise multi-precision coordinates and geometry-conditioned spectral attenuation to reduce sensitivity to inaccurate projected addresses. Local patch memory supplies view-specific appearance, while recurrent global memory provides scene context. Footprint-guided retrieval selects historical patches under a fixed local read budget. Under matched retrieval and read budgets, GLIMPSE improves geometric correspondence and photometric consistency over deterministic positional transport.