Future-informed method improves vision-language navigation success rates
FINE: Future-Informed Navigation Encoding for Data-Efficient Vision-Language Navigation
RoboticsComputer Vision and Pattern Recognition
Summary
Teaching robots to follow spoken directions in new places is expensive because they need many examples to learn well. The authors created a method called FINE that helps the robot understand which landmarks it will see ahead in the route by using information about the future from existing training data. This approach improves navigation accuracy, especially when there are fewer training examples available. As a result, the robot can learn to follow instructions more efficiently, encountering less performance loss when fewer demonstrations are provided.
What this means in practice
- •For robotic system developers: Enhance embodied robot navigation systems to better learn routes from fewer instruction demonstrations by using future-informed landmark representations.
- •For augmented reality engineers: Improve AR applications that require following navigation instructions by integrating future landmark encoding to reduce reliance on large training datasets.
Authors
Khang H. Nguyen, Hoang Pham Quang Nguyen, Ha Phuong Nguyen, Khanh Dinh Binh, Xuan Ha Nguyen, Vien Ngo, Duy Ho Nguyen Minh, Huan Nguyen, An T. Le
Abstract
Adapting vision-language navigation (VLN) policies to new environments is expensive because every additional route and instruction requires an embodied demonstration. Yet standard observation-to-action training uses only a small fraction of the information already contained in each trajectory. In particular, future observations reveal the instruction-relevant landmarks that the agent will encounter, including what they look like and how they are arranged in 3D. We introduce FINE, a Future-Informed Navigation Encoding framework that extracts this latent supervision from existing demonstrations. FINE equips a VLN backbone with two complementary auxiliary representations. First, explicit landmark tokens follow the ordered landmarks specified by the instruction and are trained to predict future landmark regions in both semantic 2D patch-feature space and viewpoint-dependent 3D geometric feature space. Second, an implicit future token learns to distinguish the landmark state that is actually reached from plausible same-scene counterfactual futures generated by a video world model. On R2R-CE and RxR-CE val-unseen, FINE improves InternVLA-N1 by 2.6 and 4.5 success-rate points, respectively, at full training data. More importantly, as demonstrations become limited, the benefit grows: at a 70% demonstration budget, FINE improves success rate by 6.8 points, recovering roughly one-third of the performance lost by reducing the training demonstrations. Project page is available at https://finevln.github.io/.