Current 4D foundation models struggle to remember moving objects
Can 4D Foundation Models Remember?
Computer Vision and Pattern Recognition
Summary
Remembering what we see in a changing environment is important for interacting with the world. The authors studied how well current 4D models can remember objects after they move out of sight. They created PersistBench, a new test using all-around videos to check if models keep track of objects, their movements, and looks over time. Their tests show that existing models only keep short-term memories and quickly lose track once objects disappear from view.
What this means in practice
- •For robotics engineers: Improve robot perception systems by identifying weaknesses in visual memory when objects leave their camera view.
- •For augmented reality developers: Develop better AR experiences by understanding limits in tracking objects that move out of sight and reappear.
Authors
Guangzhao He, Hadar Averbuch-Elor, Wei-Chiu Ma
Abstract
Perceiving and remembering the visual world is fundamental to navigating and interacting with our environment. Current 4D foundation models, such as camera-controllable video models or 4D reconstruction models, can perceive and reconstruct dynamic environments, but how well they remember what they have perceived remains an open question. Existing benchmarks largely rely on pixel-level metrics and lack ground truth for objects once they leave the field of view, making them unable to evaluate visual memory in an object-centric manner against references. To fill this gap, we introduce PersistBench, a dataset and metric suite that leverages 360° videos as omniscient ground truth and proposes three evaluation aspects: object permanence, motion continuity, and appearance preservation. Evaluating various models across diverse categories reveals that current models can only maintain short-term consistency that degrades significantly once objects leave the field of view. Our findings highlight the gap between current model capabilities and robust visual memory ("seeing is not remembering"), providing guidance for future development of 4D foundation models. Dataset and code are available on the project page: https://guangzhaohe.com/persistbench.