Vision and language navigation improves with human-guided error recovery
AVERT-VLN: Abstention-aware Visual Error Recovery and Training for Vision-and-Language Navigation
Artificial Intelligence
Summary
Robots that follow language instructions to move around new places often get lost because they don't know the layout well. The authors created a system that watches the robot's actions and can tell when it makes a mistake. When it spots a problem, it stops and asks a human for help to get back on track. The system then learns from these corrections to do better in the future without needing constant human help.
What this means in practice
- •For robotics developers: Improve robot navigation in new environments by integrating a system that requests human help only when errors are detected and learns from corrections.
- •For industrial automation teams: Use human-assisted error recovery to increase success rates of robots navigating complex spaces with language instructions.
- •For assistive technology developers: Create adaptive tools that guide visually impaired users by detecting navigation errors and seeking corrective input to improve routes.$Commercial implications: Enables new assistive navigation products that combine autonomous guidance with human correction to improve reliability.
Authors
Minrui Liu, Jingke Wang, Yuehao Huang, Hao Su, Jiajun Lv, Yukai Ma, Yong Liu
Abstract
Deploying vision-and-language navigation (VLN) agents in unseen environments remains challenging because unfamiliar layouts and visual conditions can cause execution to go off track. Rather than relying on continuous human supervision, a practical strategy is to selectively request corrective guidance, recover the ongoing task, and reuse corrective interactions to improve subsequent navigation. We propose Abstention-aware Visual Error Recovery and Training for Vision-and-Language Navigation (AVERT-VLN), a closed-loop framework that uses a plug-in vision-language Monitor for online human-assisted recovery and offline preference learning. The Monitor operates separately from navigation decision generation and assesses instruction-execution consistency from the instruction, visual history, and current observation. To train the Monitor for deviation recognition, we construct LOSTNAV DATASET with 20K counterfactual risk trajectories and rule-based deviation labels. The Monitor is first fine-tuned on 40K normal trajectories to assess instruction progress and then jointly fine-tuned on normal and risk trajectories to recognize semantic deviations. At runtime, Asynchronous Sidecar Monitoring evaluates execution alongside the navigation model. When the controller accepts a LOST verdict, it suspends autonomous execution and requests human guidance for recovery. For offline policy improvement, Trajectory-Anchored Preference Learning converts deviation-associated failures into decision-level preference pairs under shared decision contexts, restricting supervision to the decisions targeted for correction. Under human-assisted evaluation, the full AVERT-VLN system achieves success rates of 76.2% and 66.3% on the val-unseen splits of R2R-CE and RxR-CE, respectively. The same monitoring and human-assisted recovery interface also improves success rates across the three evaluated navigation architectures.