GeoLatent improves 3D spatial reasoning from 2D images using guided latent structuring
GeoLatent: Geometry-Guided Latent Structuring with Routed Optimization for 3D Reasoning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding three-dimensional space from flat images can be tricky for computers because space is continuous and complex. The authors developed GeoLatent, a method that separately represents different geometric elements like position and direction to better model 3D scenes. Their approach uses a special technique to keep these representations rich and diverse during training, which helps the system answer questions about spatial layouts more accurately. They tested this method on standard benchmarks and found it outperforms previous techniques.
What this means in practice
- •For computer vision engineers: Create more accurate AI systems that interpret 3D layouts from 2D images for spatial question answering tasks.
- •For robotics software developers: Enhance robot perception modules to better understand and interact with 3D environments using 2D vision inputs.
Authors
Yakun Zhu, Yi Bin, Yujuan Ding, Zheng Wang, Pengpeng Zeng, Duo Peng, Jingkuan Song, Heng Tao Shen
Abstract
Despite progress in vision-language models, 3D spatial reasoning from 2D images remains challenging. Text-based methods describe intermediate geometry with discrete tokens, limiting fidelity for continuous spatial relations. Continuous latents offer richer representations, but a single latent type does not explicitly separate the cues needed across spatial tasks. Decomposed spatial latents address this by representing position, direction, and global geometry separately under geometric supervision. Yet the geometry representation can still collapse toward one dominant direction, and unrestricted attention can leave the latents underused during answer learning. We introduce GeoLatent, combining Common--Residual Geometry Alignment (CR-GEO) with routed optimization to structure the geometry states while promoting latent-mediated answer learning. CR-GEO separates shared from residual teacher geometry; routed optimization jointly trains geometry and language, temporarily directs visual answer learning through the latents, and restores full attention with geometry supervision. In controlled comparisons, CR-GEO raises geometry effective rank from 1.00 to 3.87, while blocking latent readout at the bottleneck lowers direction accuracy from 89.1% to 25.8% on 128 fixed questions. After recovery, the differentiated geometry representation and latent-mediated visual route remain available alongside direct image access. GeoLatent achieves 73.0% on SPAR-Bench and 72.1% on SPBench, outperforming previously reported methods on both.