Vision and language navigation improves with targeted human feedback
Asking Less, Adapting Better: Feedback-Efficient Test-Time Adaptation for Vision-and-Language Navigation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Following instructions to move through unfamiliar places is hard for AI because new environments can confuse them. The paper's authors found that letting the system adapt itself during use with selective help from humans works better than always asking for feedback. They created a way to pick exactly when and where to ask for human input, so the AI only gets help at important moments. This method led to better navigation with fewer interruptions and still worked well even if the human feedback was sometimes wrong.
What this means in practice
- •For robotic navigation teams: Improve robot navigation in new environments by efficiently incorporating human corrections during operation to reduce errors and failures.
- •For virtual assistant developers: Enhance virtual assistants' ability to follow complex instructions and adapt on the fly by selectively requesting user feedback only when necessary.
Authors
Zixu Wang, Huaquan Zhang, Shoukai Xu, Jinwu Hu, Zhiquan Wen, Mingkui Tan
Abstract
Vision-and-Language Navigation (VLN) requires agents to follow instructions and navigate to target locations. However, current VLN agents often suffer severe performance degradation in unseen environments due to distribution shifts. Test-Time Adaptation (TTA) offers a promising paradigm to bridge this gap, yet existing methods that rely on unsupervised entropy minimization can propagate errors through the trajectory, causing cascading failures in navigation. Recent methods mitigate this by incorporating human feedback, but they apply trajectory-level supervision uniformly across all steps without localizing the actions is truly informative, leading to inefficient feedback utilization and mismatched optimization granularity. To address these limitations, we propose FETA, a Feedback-Efficient Test-Time Adaptation method that improves both feedback efficiency and optimization reliability. To reduce unnecessary human interventions, we introduce a hierarchical sample selection strategy to localize informative samples. Specifically, we first filter trajectories whose normalized policy entropy exhibits large fluctuations. Within each selected trajectory, we further select the critical action via an entropy-gradient saliency criterion and query a single binary feedback on its correctness. A discrepancy-aware entropy optimization objective then updates the policy only at the queried step, with adaptive scaling that assigns larger relative weights to samples with greater prediction-feedback discrepancy. Extensive experiments across multiple benchmarks demonstrate that FETA achieves state-of-the-art performance with fewer human interventions, while remaining robust under noisy feedback and continual adaptation settings.