AI summaryⓘ
The authors created MultiView-Bench, a new test for vision-language models (VLMs) to see if they can understand 3D scenes by combining views from different angles. Unlike past tests, this one checks if models can place objects in a fixed global space, not just relative to one camera view. Their evaluation showed that current models do well with simple 2D relations from one image but struggle with 3D spatial understanding and merging information across multiple views. They also found biases like difficulty with unusual view directions and changes in color or texture. To improve this, the authors proposed ViewNavigator, a system that smartly picks viewpoints and combines information to help models perform better on this challenge.
Vision-Language ModelsMulti-View Integration3D Scene ComprehensionAllocentric Coordinate SystemBenchmark DatasetSpatial RelationsViewpoint SelectionObject PositioningMulti-Agent FrameworkModel Bias
Authors
Hantao Zhang, Jinru Sui, Ed Li, Dirk Bergemann, Zhuoran Yang
Abstract
Recent benchmarks for VLMs largely assess single- or limited-view perception, leaving untested the core cognitive ability to integrate observations across viewpoints into a coherent, world-centric (allocentric) 3D mental model. We introduce MultiView-Bench, a diagnostic benchmark expressly designed to evaluate multi-view integration for holistic 3D scene comprehension. Unlike existing datasets that focus on pixel-level mapping or camera-relative navigation, MultiView-Bench requires models to decouple object positioning from transient perspectives and ground them in a fixed global coordinate system. This capability serves as a prerequisite for VLMs before being deployed for downstream tasks such as mechanical part assembly. Our systematic evaluation of frontier VLMs reveals consistent failure modes: strong performance on 2D planar relations from a single image, but marked difficulty with 3D spatial relations and with aggregating information across views. We further identify biases in VLMs, such as struggles with unconventional axis directions and sensitivity to object colorways and texture variations. Acknowledging these limitations, we propose ViewNavigator, a multi-agent framework that actively selects informative viewpoints, perceives, and fuses multi-view evidence, improving diverse base models on MultiView-Bench even under a strict budget-matched comparison (and by 3-5x for the full agent).