SuperNav enables robots to navigate any scene for any task
SuperNav: An Agentic Navigation System for Any Task in Any Scene
RoboticsComputer Vision and Pattern Recognition
Summary
Robots that help around the house or workplace need to move around in new places and handle many different tasks. The authors designed SuperNav, a system where a large language model understands what to do and where to go, but delegates the actual moving part to specialized navigation tools. This makes the robot better at handling new environments and requests without extra training. They showed SuperNav works well in both simulations and on a real robot.
What this means in practice
- •For robotics engineers: Develop navigation systems that can understand diverse tasks and adapt to new environments without retraining the core language model.
- •For field robotics teams: Deploy quadruped or mobile robots capable of navigating unfamiliar indoor scenes to perform user-specified tasks.
- •For warehouse automation developers: Integrate flexible navigation agents into robots for complex multi-object retrieval and demand-driven tasks in large warehouse settings.$Commercial implications: This system allows selling adaptable robotic automation solutions that handle variable tasks and environments without costly retraining.
Authors
Jinkai Zhang, Jingyi Xu, Yuanhong Yu, Jiarui Guo, Ruizhen Hu, Hujun Bao, Xiaowei Zhou, Sida Peng
Abstract
General-purpose service robots need navigation systems that can handle diverse human requests in unfamiliar environments, combining task generality with scene generality. Some existing methods fine-tune multimodal large language models (MLLMs) to predict navigation actions, making their behavior dependent on the coverage of navigation training data and potentially limiting generalization to new requests and environments. Our key insight is to let the MLLM focus on interpreting requests, understanding scenes, and making decisions while preserving its general-purpose capabilities and delegating motion execution to navigation tools. To realize this idea, we introduce SuperNav, which equips a pretrained MLLM with a specialized agent harness without navigation-specific fine-tuning of the MLLM. Our harness supports these decisions with Navigation Skills, agent-oriented Tools for physical interaction, and task-progress and context management. A unified visual-point interface connects decision-making to motion by allowing the model to specify destinations directly in images and revise its decisions from execution feedback. Together, these components support sustained navigation across different task requirements and environments. SuperNav outperforms four evaluated baselines on instance-level, multi-object, and demand-driven tasks. Category-level evaluation on HM3D and deployment on a real quadruped robot further demonstrate its applicability across environments. Project Page: https://zju3dv.github.io/SuperNav/