Vision guided robot navigation learns to plan and act together

MATCH: Execution-Aware Hierarchical Reinforcement Learning for Vision-and-Language Navigation in Continuous Environment

Robotics

Summary

Robots that navigate by following spoken or written instructions need to both plan routes and avoid bumps or obstacles while moving. This paper presents MATCH, a method that trains a robot’s planning and movement parts together, so the robot can better understand instructions and move more safely and efficiently. The authors show that MATCH works well in simulated tests and even in experiments with real robots, proving its promise for practical use.

What this means in practice

  • For autonomous robotics teams: Improve robot navigation by linking route planning with real-time movement control for instruction-following tasks in real environments.
  • For robot software developers: Develop robot control software that adapts planning commands based on how an executor performs, enhancing navigation reliability.

Authors

Shuhao Ye, Yufei Wei, Demao Wang, Shichao Zhai, Rong Xiong, Yue Wang

Abstract

Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires instruction-consistent planning and reliable execution under collision constraints. Yet many hierarchical approaches optimize the planner while keeping execution fixed. We introduce MATCH, a framework that jointly learns a language-conditioned planner and a goal-conditioned primitive-action executor. An execution-aware topological graph represents start-dependent execution options and integrates executor value estimates with realized outcome memory, enabling the planner to select execution requests and refine their targets through bounded continuous offsets. A transition-aware dense macro reward balances ordered progress along the reference route against execution residual, encouraging executable requests without substituting local success for instruction following. A staged curriculum culminates in synchronized on-policy updates, with the executor learning on planner-generated requests and the planner adapting its requests to the executor's current behavior. With 0.5B parameters, MATCH achieves state-of-the-art performance on the R2R-CE and RxR-CE benchmarks. Preliminary real-robot deployments further demonstrate its practical potential.