Vision language navigation improved by matching physical action feasibility
Revealing and Bridging the Physical Executability Gap in Vision-Language Navigation
Robotics
Summary
Vision-language navigation systems help robots follow instructions to reach goals, but what looks good in a computer simulation may not work in the real world because robots can’t always perform some actions physically. The authors found this gap between understanding instructions and being able to move physically in existing simulators. They created a new method called VeSTA that guides robots to predict paths that are not only semantically correct but also physically possible and safe. Their experiments show that VeSTA reduces collisions and works well both in simulation and in real robot tests.
What this means in practice
- •For robotics engineers: Improve robot path planning to ensure generated navigation actions are safely executable in physical environments.
- •For autonomous vehicle developers: Use the approach to reduce simulation-to-reality gaps by aligning predicted routes with physically feasible trajectories.
Authors
Gang He, Zhile Cheng, Kepeng Xu, Zhenyang Liu, Yu-Wing Tai
Abstract
Continuous vision-and-language navigation (VLN) is typically evaluated by goal-reaching success, assuming it reflects an agent's ability to execute language-grounded behavior. We show that this assumption can fail: permissive contact models in current simulators may mask physically infeasible actions, creating a physical executability gap between semantic competence and embodied realization. To bridge this gap, we introduce Visual-Semantic Traversability Alignment (VeSTA), a framework that aligns VLN policies with physically executable futures. We define visual-semantic traversability as the ability to predict instruction-conditioned trajectories that are semantically valid, geometrically feasible, and physically robust. VeSTA addresses this misalignment through two complementary mechanisms: improving executable support in trajectory generation and risk-aware selection among multimodal futures. Specifically, Trajectory Relative Group Policy Optimization uses strict-contact feedback to reshape the candidate distribution toward executable regions, while the Traversability Risk Critic learns a risk-adjusted scalar utility for candidate-wise future selection. Experiments on R2R and RxR reveal substantial dependence of existing VLN agents on simulator contact semantics. Under strict-contact evaluation, VeSTA substantially reduces collision-induced failures while preserving or improving goal-reaching performance, and controlled real-world robot experiments provide evidence that the learned preference transfers beyond simulation.