Visual imagination methods beyond RGB improve spatial reasoning accuracy
Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Current methods that imagine what unseen places look like mostly create color pictures, but these aren’t always the best for solving spatial questions. The authors show that different tasks need different kinds of imagined pictures, like maps or panoramic views, to do better. They made a new test set called ImagiRep to compare these visual types directly, and found that sometimes color images can actually mislead. Finally, they created a method called GEARS that picks the best kind of image for each question, improving how machines reason about places they haven’t seen.
What this means in practice
- •For robotics developers: Improve robots’ understanding of unseen spaces by selecting visual representations that better support spatial decision-making beyond standard color images.
- •For game environment designers: Use varied imagined spatial views to enhance NPC navigation and strategy by providing multiple perspectives of unexplored game areas.
Authors
Zengqi Zhao, Zun Wang, Shoubin Yu, Mohit Bansal, Yue Zhang
Abstract
Visual world models increasingly support spatial reasoning by generating RGB views of unobserved states. However, RGB has become the default interface largely because it is well suited for visual generation, rather than necessarily being the most effective representation for downstream reasoning. Different spatial questions may benefit from different representations of the same imagined state: top-down geometry can facilitate global distance comparisons, while panoramic views provide broader context for reasoning about open space. To isolate the role of representation in spatial imagination, we introduce ImagiRep, a simulator-grounded benchmark of 922 questions spanning seven question types across two settings, Viewpoint Imagination and Cross-view Integration, providing oracle RGB and six alternative representations (e.g., depth, panoramic views, and top-down geometry) of the same unobserved state. This controlled comparison reveals that even accurate imagined RGB does not guarantee accurate spatial reasoning, and different questions benefit from different representations. Similar trends hold for representations derived from world-model-estimated states. Notably, estimated RGB alone can mislead spatial reasoning, while alternative representations retain complementary benefits. Building on these findings, we introduce GEARS, a lightweight test-time strategy for adaptive representation selection, outperforming existing spatial imagination methods and showing that reasoning about unseen states requires more than generating RGB views.