Vision language models get better at explaining navigation steps
Explaining Navigation Decisions: Stage-Aware Visual Attribution for Vision-Language Navigation
Computer Vision and Pattern Recognition
Summary
Robots that follow spoken or written directions need to explain how they decide their next moves, especially when those moves depend on different parts of the instructions over time. The authors found that existing explanation methods often miss this timing aspect and give similar reasons even when the instructions change. They created a new method that links robot decisions to specific stages in the instruction by combining past and current visual clues and filtering out unrelated details. Their approach works better across different navigation setups and can help fix mistakes or run robots in real environments.
What this means in practice
- •For robotics developers: Improve transparency of robot actions by clarifying which parts of instructions and visual input lead to navigation decisions.
- •For autonomous vehicle testers: Diagnose and analyze navigation failures by connecting visual cues to specific instruction stages during autonomous driving tests.
Authors
Yiyuan Yao, Xinpeng Ding, Minglei Chen, Fei Gao, Lihuo He, Jie Yuan, Aijun Li, Nannan Wang, Xinbo Gao
Abstract
As Vision-Language Models (VLMs) increasingly enable robots to follow language instructions, understanding their navigation decisions becomes important for reliable deployment. Existing explainability methods overlook a key challenge in Vision-Language Navigation (VLN): evidence for the current action may be distributed across historical and current observations and associated with different instruction stages. We identify instruction-stage insensitivity, where existing methods produce similar explanations despite changes across different navigation stages. To address this issue, we propose a stage-aware attribution framework for VLN. We first construct action-conditioned temporal instruction--visual attributions across observations. These attributions are projected into instruction space to estimate stage associations, which guide the aggregation of relevant historical evidence for refining the current explanation. An instruction-guided spatial filter further suppresses isolated responses while preserving compact, instruction-relevant regions. Experiments across multiple VLN policies and benchmarks show improved attribution quality over existing methods. We further demonstrate that our method acts as a diagnostic tool for analyzing failure cases, and validate its applicability to real-world navigation on a Unitree G1 robot.