Vision and language navigation agents get reliable uncertainty estimates
ENCP: Episode-Normalized Conformal Prediction for Vision-and-Language Navigation
Machine LearningArtificial IntelligenceRobotics
Summary
Vision-and-language navigation agents help robots follow instructions to move around, but they sometimes make uncertain or wrong guesses about what to do next. The authors developed a new way called Episode-Normalized Conformal Prediction (ENCP) that better measures how confident the agent is at each step in a sequence of actions. Their method guarantees the agent's uncertainty estimates cover the true position with high probability, even though the steps depend on each other. This helps the agent decide when to ask for help or take safer actions.
What this means in practice
- •For robotics engineers: Improve navigation robot safety by reliably flagging uncertain steps needing human override.
- •For autonomous vehicle developers: Integrate stepwise uncertainty measures to make navigation decisions more robust and interpretable in complex environments.
Authors
Vicky Feliren, A. Taufiq Asyhari, Muhamad Risqi U. Saputra
Abstract
Uncertainty estimation for Vision-Language-Navigation (VLN) models is a critical task since it can help identify ambiguous and unreliable predictions, enabling agents to make safer navigation decisions. As one of the most advanced uncertainty estimation frameworks, conformal prediction (CP) offers a promising approach for uncertainty estimation in VLN. However, given that VLN agent requires a sequence of steps, standard calibration in conformal prediction fails to provide coverage guarantee it promises over a dependent, variable-length VLN episode. To this end, we propose Episode-Normalized Conformal Prediction (ENCP), which rescales a nonconformity score by the policy's residual confidence and calibrates one maximum score per episode. Under exchangeable calibration and test episodes, this construction covers the ground truth at every step with probability at least $1 - α$, while allowing dependence among steps within an episode. Across four VLN policies and three nonconformity scores on R2R and REVERIE dataset, ENCP meets all reported empirical step-coverage targets on the seen-to-unseen evaluation. These results demonstrate that ENCP can provide model-agnostic uncertainty estimates, which might be useful for determining when a VLN agent should defer to a more capable predictor, including human assistance.