Vision language models struggle with viewpoint based spatial reasoning
Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models
Computer Vision and Pattern RecognitionComputation and LanguageMachine LearningRobotics
Summary
Understanding instructions about space from another person's viewpoint is difficult for computer systems that combine images and language. The authors created a new test called POVBench to see how well these systems understand directions when the speaker's perspective is important. They found that current models still have trouble locating objects based on directional language, even when told exactly how to consider the speaker's viewpoint. Breaking down the task into smaller parts related to the observer's perspective helps improve performance.
What this means in practice
- •For robotics engineers: Improve robot understanding of human spatial instructions by testing and refining models that interpret directions from a speaker's viewpoint using the POVBench dataset.
- •For augmented reality developers: Enhance AR systems to better interpret spatial commands in 3D environments by integrating vision-language models evaluated on situated spatial reasoning tasks.
Authors
Mimo Shirasaka, Haochen Zhang, Yonatan Bisk
Abstract
Reasoning over language instructions in embodied tasks such as robotics often requires understanding spatial relations from a speaker's situated perspective. Humans infer such perspectives from shared environmental knowledge, activity context, and commonsense. Recent vision-language models (VLMs) appear capable of spatial reasoning, but their ability to infer a speaker's viewpoint from contextual cues and interpret situated spatial relations from that viewpoint remains unclear. We call this capability contextual observer grounding. To study this capability, we construct the Point-of-View Benchmark (POVBench), a dataset of 3D scenes and queries that disentangles Inferred, Stated, and Given forms of observer grounding in natural embodied communication. Given multi-view observations and a natural-language sentence, models must localize unseen or underspecified targets from situated spatial and contextual cues. Across state-of-the-art VLMs, localizing targets from directional language remains challenging, even when observer grounding is made explicit. We find that explicit breakdowns of observer-relative spatial reasoning improve target localization. Our project page is available at https://mimo-owl.github.io/POVBench/.