Visual foundation models show mixed ability to track and update spatial info
Probing Spatial State Representations in Visual Foundation Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
People and robots need to keep track of where things are, even when they can’t see everything at once. The authors tested if powerful visual AI models can remember and predict spatial layouts over time and use that to act toward goals. They found these models can recognize what they currently see well but struggle to keep track of objects when out of sight or to predict future views accurately. Also, using this information to take goal-directed actions doesn’t always improve performance. This study shows that just reading what a model sees isn’t enough – we need to check if it can remember, update, and use spatial info effectively.
What this means in practice
- •For robotics engineers: Improve robots’ ability to track objects over time and predict future scenes for navigation in indoor environments.
- •For augmented reality developers: Enhance AR systems to maintain spatial awareness beyond current views, improving persistent virtual overlays indoors.
Authors
Baiqiao Yin, Zhongang Cai, Ruisi Wang, Qiwei Liang, Lei Yang, Ziwei Liu
Abstract
An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We study this question through four levels of spatial capability---spatial structure, object persistence, predictive spatial update, and actionability---using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments. The resulting capability profiles vary across tasks, network layers, and prefix lengths: strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. Together, these findings motivate evaluating spatial representations not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.