AdaGeoVLN improves vision-language navigation using geometry memory
AdaGeoVLN: Selective Geometry Across Representation Depth and Navigation Time for Vision-Language Navigation
Robotics
Summary
Vision-language navigation means following instructions while recognizing what you see and where you are. This paper introduces AdaGeoVLN, a system that uses different layers of 3D scene information and remembers important past views during navigation to make better decisions. The system carefully manages what geometric details it keeps in memory, keeping navigation efficient without needing extra data. Experimental tests show it performs well on standard navigation tasks using only regular RGB images.
What this means in practice
- •For robotics engineers: Enable robots to navigate physical spaces by integrating layered 3D geometry cues and relevant past observations efficiently.
- •For augmented reality developers: Support AR devices in following spoken navigation instructions using a single RGB camera and maintaining spatial awareness over time.
Authors
Quan-Dung Pham, Anh Dao, Danh Vinh Le, Nguyen Viet Tri Pham, The-Anh Nguyen, Zhirui Dai, Yiyu Chen, Tuyen P. Le, Truong Nguyen, Quan Nguyen
Abstract
Vision-language navigation requires aligning language with visual observations while maintaining spatial understanding over time. Geometry foundation models (GFMs) expose intermediate representations throughout their hierarchy, but how navigation policies should use these features and retain historical geometric evidence remains unresolved. We introduce \method{}, a streaming VLN framework that addresses these questions across \textbf{representation depth} and \textbf{navigation time}. Hierarchical GFM--VLM fusion couples earlier, intermediate, and later GFM representations to successive policy stages instead of repeatedly injecting a terminal feature. Navigation-aware GFM memory retains historical VGGT global-attention KV states according to instruction relevance, geometric confidence, and transition novelty under a bounded per-layer budget. Retained states provide geometric context for subsequent observations before fusion with the policy. Across R2R-CE and RxR-CE, \method{} achieves strong performance using a single RGB stream without additional navigation-specific external data. Controlled ablations show that multi-depth coupling substantially outperforms repeated terminal-feature injection at matched fusion locations. Bounded navigation-aware retention preserves navigation performance while considerably reducing GFM-KV memory relative to larger-memory temporal retention. These findings support jointly examining the geometric representations exposed to the policy and the historical evidence retained for future inference. Code will be released upon acceptance at https://humanoid-research.github.io/adageovln/.