Anticipate Before Acting: Future-State-Conditioned Vision-Language Navigation
2026-07-20 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors study how robots following instructions and seeing their environment can better decide what to do next by thinking about the future. They first show that if the robot gets a peek at future images from an expert path, it chooses better actions, but those images aren’t available in real use. To fix this, they teach the robot to predict a compact representation of future visuals during training, which helps it navigate better when it doesn’t actually see the future images. Their method, called FSC-VLN, improves performance on a navigation benchmark by conditioning decisions on future visual information learned only during training.
vision-language navigationbehavior cloningfuture visual latentegocentric observationspolicy stateR2R benchmarkfuture-query tokentraining supervisionaction predictionembedding alignment
Authors
Lingfeng Zhang, Zhanguang Zhang, Liheng Ma, Tongtong Cao, Yingxue Zhang
Abstract
End-to-end vision-language navigation (VLN) with causal vision-language models can map instructions and egocentric observations directly to actions, but standard behavior cloning supervises only the next action and does not explicitly train the policy state to be predictive of future visual outcomes. We first ask a diagnostic question: if the policy is given an expert-trajectory future image as privileged input at training and testing time, is that additional visual evidence useful for choosing the current action? (These expert-trajectory future images are unavailable at test time in real deployment, so we use this setting only as a privileged-input diagnostic.) The answer is yes; this sanity check shows that future observations can provide rich, actionable cues. We then ask a deployable question: without accessing future images at inference, can we still benefit from future information by using a compressed future visual latent only as training supervision? We propose Future-State-Conditioned VLN (FSC-VLN), which adds a future-query token and aligns its hidden state to a frozen visual embedding $Δ$ steps ahead via a training-only target branch that is removed after training. On R2R val-unseen, FSC-VLN improves SR/OSR/SPL over a StreamVLN-style baseline under two training-data regimes, with larger gains on long-horizon episodes; ablations further support the dual-query design (separating future and action queries).