Embodied navigation agents improve zero-shot tasks with agent harness
HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness
Robotics
Summary
Navigating indoor spaces with instructions or object goals is hard for robots because they need to see, remember places, and decide what actions to take. The authors designed HarnessVLN, a system that doesn't need training to work and uses a central 'Agent Harness' to check plans against what the robot sees and remembers. This harness helps the robot recover from mistakes and keeps track of progress, enabling more reliable navigation. The system works well across several benchmarks and even with humanoid robots in real environments.
What this means in practice
- •For robotics engineers: Build robots that navigate indoor spaces using natural language without retraining for new environments, improving reliability by verifying plans with spatial memory.
- •For warehouse automation teams: Use zero-shot navigation methods to reconfigure robots on-demand for different indoor object retrieval tasks without additional training cycles.
Authors
Yang Chen, Lirong Che, Zhenyu Huang, Wenbo Fu, Chuang Wang, Xu Cao, Daqi Liu, Yuzhe Yang, Jian Su, Lan-Zhe Guo
Abstract
Embodied navigation requires agents to interpret visual observations, accumulate spatial knowledge, and execute actions to follow instructions or locate objects. Training-based methods face generalization challenges, while training-free methods exploit multimodal large language models (MLLMs) but often lack mechanisms to reconcile proposed actions with spatial evidence, task progress, and execution failures. We present HarnessVLN, a zero-shot, training-free framework whose Agent Harness coordinates perception, retrieval, grounding, navigation, recovery, and termination through a unified tool interface. The Harness validates planner proposals against spatial evidence, geometric feasibility, and subgoal consistency, incorporating structured tool feedback into subsequent decisions. Hierarchical event memory tracks task progress and execution history, while a persistent Spatiotemporal Graph maintains reusable spatial evidence and failure annotations for verification and recovery. A replaceable Navigation Executor converts validated targets into executable motions, allowing the same Harness protocol to support instruction-following and object-goal navigation. HarnessVLN achieves success rates of 60.8%, 53.9%, 76.0%, and 59.3% on R2R, RxR, HM3D-v2, and HM3D-OVON, respectively, surpassing prior training-free SOTA results. Humanoid deployment further demonstrates its applicability to both tasks in real-world environments. The project page is: https://harnessvln.netlify.app/.