3D-Aware VLMs with Implicit and Explicit Geometries
2026-07-23 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors created VLM-IE3D, a vision-language model that improves understanding of 3D space using only RGB videos. They combine two types of 3D information: Implicit Geometry Tokens for general 3D shape ideas, and Explicit Geometry Tokens for detailed 3D structures. Their method integrates these 3D cues with 2D visual data to help the model better handle tasks requiring spatial reasoning. Tests show their approach works well on several 3D tasks without needing extra 3D inputs.
vision-language models3D spatial reasoningRGB videosImplicit Geometry TokensExplicit Geometry Tokens3D video detection3D visual groundingdense captioning3D-aware adapter
Authors
Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Ran Xu, Shijian Lu, Gongjie Zhang
Abstract
Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.