Exposing the Long-tail in Embodied Urban Navigation via Scalable Learning from In-the-Wild Videos
2026-08-17 • Robotics
Robotics
AI summaryⓘ
The authors created a system to teach robots how to navigate cities by using lots of real-world videos from the internet. Instead of collecting expensive and limited data, their method automatically labels these videos with locations and navigation instructions. They train a model to understand and plan routes by combining what it sees with language commands. The authors also study where the model makes mistakes, especially on unusual or rare situations, to improve it. Tests show their approach helps robots learn better navigation from everyday videos and reveals detailed insights about hard problems in navigation.
embodied navigationegocentric videospoint-goal navigationvision-language-action policytrajectory annotationurban navigationlong-tail distributionknowledge transferfailure mode analysisreflection-based analysis
Authors
Bingyi Xia, Han Bao, Zhewei Chen, Hanjing Ye, Jingwen Yu, Yuhan Pang, Wenjun Xu, Jiankun Wang
Abstract
Learning embodied urban navigation policies from real-world data is constrained by the cost of task-specific data collection and the limited coverage of rare yet safety-critical scenarios. To address these challenges, we present a scalable framework for learning point-goal urban navigation from web-scale in-the-wild egocentric videos while systematically exposing its long tail. The framework automatically annotates uncurated web videos with metric trajectories and structured navigation semantics, which are then used to train a vision-language-action policy for interpretable navigation planning. We characterize the long tail based on model performance and the distribution of perception-motion patterns, and employ reflection-based analysis to diagnose recurring failure modes. Experiments on web-video data and real-world urban navigation tasks demonstrate effective knowledge transfer from unconstrained videos and reveal coherent long-tail structures beyond aggregate navigation performance.