Long-range robot navigation improves by combining scene understanding and geometry
MOSU2: Long-range Navigation System with Multi-modal Scene Understanding
Robotics
Summary
Robots moving through public outdoor spaces need to safely navigate around people, vehicles, and various obstacles while staying on suitable paths. The authors show that using a two-step system helps robots decide when to get more detailed information about their environment, combining visual cues with measurements of distance and motion. This helps the robot better understand how scenes change over time, like whether a car is slowing down or a cyclist is about to cross. Their system, MOSU2, improves navigation success and accuracy compared to prior approaches relying on single images.
What this means in practice
- •For robot navigation teams: Improve outdoor robots' long-distance navigation by combining visual scene context with geometric and motion data for safer path planning.
- •For autonomous vehicle engineers: Integrate multi-modal scene understanding to better interpret interactions with pedestrians and cyclists in complex outdoor traffic environments.
Authors
Jing Liang, Zhenyu Hou, Senthurbavan Kirubaharan, Md Safwan Mondal, Ashok Gudivada, Xuesu Xiao, Dinesh Manocha
Abstract
Robots that travel long distances through public outdoor spaces must do more than avoid obstacles: they must remain on terrain suited to their mobility and decide when to proceed, yield, or detour around pedestrians, cyclists, vehicles, traffic signals, and temporary restrictions. These decisions are difficult because their correctness often depends on how a scene evolves, such as whether an approaching car is slowing or a cyclist will cross the robot's path. Vision-language models (VLMs) can interpret such cues, but they reason over images that lack metric geometry and motion. Supplying this evidence at every step is too slow for real-time navigation, and hand-crafted triggers cannot anticipate the open-ended variety of outdoor situations. We present MOSU2, a long-range navigation system with two-stage trajectory selection, in which first-stage contextual reasoning determines when additional evidence is needed and what to retrieve. Only when an interaction remains unresolved does the first stage trigger the second stage to retrieve geometric and dynamic information and align it with scene context to make navigation decisions. Building on MOSU, MOSU2 follows routes from GPS-based route planning and filters candidate trajectories through multi-modal traversability analysis before selecting a trajectory to move the robot. In real-world experiments spanning stop gestures, construction sites, crosswalks, and roads shared with vehicles, cyclists, and pedestrians, MOSU2 raises mean navigation success from 61.8% to 94.2% and condition-detection accuracy from 72.2% to 91.4% over its single-image predecessor, at about 2.2 times its mean decision latency. Project website: https://jinggm.github.io/mosu2/