Multi-agent vision and language navigation tasks with new benchmark
Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Method
Computer Vision and Pattern RecognitionArtificial IntelligenceRobotics
Summary
Following instructions to navigate a place is usually done by one robot, but many real situations need teams working together. The authors created a new way to define multi-robot navigation tasks where subtasks depend on each other and require coordination. They made a huge set of test scenarios with multiple agents and instructions, plus special ways to measure success. They also built a system called TRISS that schedules tasks, shares knowledge, and plans safe routes to avoid collisions. Their tests show TRISS sets a baseline but also that coordinating multiple robots remains a hard problem.
What this means in practice
- •For robotics teams: Coordinate multiple robots to complete complex navigation tasks where subtasks depend on each other and require safe, collision-free routes.
- •For warehouse automation developers: Use new task formulations and benchmarks to improve multi-robot coordination for picking, delivery, and resource handling in large indoor spaces.
Authors
Yunzhe Xu, Zhe Liu
Abstract
Vision-and-Language Navigation (VLN) has largely focused on a single agent following a single instruction, yet many real-world applications require teams of robots to tackle tasks beyond the capabilities of any individual agent. We present Systematic Multi-Agent Vision-and-Language Navigation, providing, to our knowledge, the first systematic formalization of multi-agent VLN as a constrained coordination problem: each mission consists of subtasks carrying dependency and resource constraints (presence locks and holding chains). A verified four-stage crafting pipeline instantiates the task as MAVLN, comprising 11,724 episodes across 145 scenes with teams of up to four agents under three instruction regimes, accompanied by tailored constraint-aware metrics. We further present TRISS, a coordination-ready navigation system coupling an LLM-based subtask scheduler, a shared topological memory that turns each agent's exploration into team knowledge, and a conflict-aware execution mechanism that realizes simultaneous intentions as collision-free routes. Extensive experiments establish TRISS as a comprehensive baseline and reveal substantial room for improvement across scheduling, planning, and execution, highlighting the challenges of coordinating under MAVLN task constraints. Project page: https://xyz9911.github.io/mavln.