ScaleNav enables robots to navigate using internet video training
ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Robotics
Summary
Robots need to find and stop at specific places described in words, but teaching them with real-world data is costly and simulated data doesn't always transfer well. The authors created ScaleNav, which learns how scenes change as a camera moves toward a goal by using lots of online egocentric videos. Then, it combines this knowledge with simulated training that includes robot actions to improve navigation. The approach works well in tests and on a real robot, helping machines understand natural directions to reach targets.
What this means in practice
- •For robotics engineers: Develop robots that understand natural language instructions and navigate diverse real environments using pretrained video knowledge combined with simulated action data.
- •For autonomous vehicle developers: Improve short-range navigation systems in autonomous platforms by training models on large-scale egocentric videos to approach language-specified targets accurately.
Authors
Yihui Yang, Yuanhong Yu, Jiawei Wang, Zhongrui Yu, Jia Shi, Kun Zhan, Ruizhen Hu, Xiaowei Zhou, Yu Liu, Sida Peng
Abstract
We study language-conditioned local navigation, where a robot must navigate to a text-specified target visible in its initial observation and stop at the described location. End-to-end navigation policies require diverse training data to generalize across scenes, yet real-robot data is costly to collect and simulated data leaves a sim-to-real gap. Internet egocentric videos offer a scalable source of real-world navigation experience. Our key insight is to learn goal-directed visual dynamics from Internet egocentric videos, and use these dynamics to support robot action generation. To this end, we propose ScaleNav, a unified world-action framework with three-stage training. We first perform instruction-conditioned video pretraining on these videos to learn how observations evolve as the camera approaches instructed targets across diverse real-world scenes. Action alignment then uses action-labeled simulated data to train the model to predict executable action chunks. Joint fine-tuning subsequently optimizes all trainable components together for navigation. ScaleNav achieves success rates of 86.5% on a single-view adaptation of Short-Horizon OVON and 94.9% on our HabitatGS-LN benchmark, outperforming the evaluated baselines on both benchmarks. We further deploy and evaluate ScaleNav on a real quadruped robot, with results supporting the model's ability to generalize to real-world navigation. Code and videos are available at https://zju3dv.github.io/ScaleNav/