Vision language navigation with monocular rgb and langgraph state control
LG-VLN: A Zero-Shot Vision-and-Language Navigation Framework with LangGraph State Orchestration
Computer Vision and Pattern RecognitionRobotics
Summary
Vision-and-language navigation means following spoken or written directions to move through a 3D space. This paper presents LG-VLN, a system that uses just regular camera images (no special sensors) and a language-based map of the task to help a robot move around. The authors designed a way to build a simple 3D map and understand instructions as steps in a graph, helping the robot decide where to go next and how to recover from mistakes. Their system works without training on the new environment and achieves reasonable success on a standard test of navigation tasks.
What this means in practice
- •For robotics engineers: Use shared visual features and language-based state control to guide robots through unknown indoor spaces using only a single camera.
- •For augmented reality developers: Enable AR systems to interpret natural language navigation instructions in new environments without needing special sensors or prior training.
Authors
Jianhe Zhao, Yanhua Qiu, Zhiyu Zhang, Zibo Zhao, Jinhua Xie
Abstract
Continuous-environment vision-and-language navigation (VLN-CE) requires interpreting natural-language instructions in unseen 3D environments and executing continuous low-level actions. Existing methods often depend on LiDAR, panoramic cameras, or extra sensors; separate geometric-mapping and semantic-navigation visual representations can cause long-trajectory spatial-semantic inconsistencies. We propose LG-VLN, a monocular zero-shot framework with shared visual features and LangGraph-based state orchestration. An online feed-forward 3D reconstruction network predicts depth, camera poses, and dense point clouds for agent-pose estimation and global map fusion. Geometry and navigation share dense CleanDIFT features: semantic consistency rejects incorrect inter-frame correspondences, while target-instance constraints define visual references whose similarity combines with local BLIP-2 image-text relevance to form a semantic value map. LangGraph represents instruction parsing, geometric perception, semantic value updates, path planning, action execution, and failure recovery as a directed state graph with conditional transitions, persistent state, and modular recovery mechanisms. On a fixed 550-episode subset of the R2R-CE val-unseen split, LG-VLN achieves 21.3% success and 12.1% success weighted by path length. Ablations show shared semantic features improve navigation, further boosted by combining visual similarity and image-text relevance. Results establish shared visual representations and explicit state orchestration as effective for zero-shot VLN-CE using monocular RGB alone. Code will be publicly released for reproducibility.