Mixed reality system improves control of multi-robot teams
MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality
Human-Computer InteractionRobotics
Summary
Coordinating teams of different kinds of robots using natural language can lead to confusion and mistakes. The authors created MRPilot, a mixed reality tool that lets people see and manage robot plans and actions more clearly. It helps users fix unclear instructions, check plans, watch robots working, and intervene if problems happen. In tests, MRPilot made users feel more confident, aware, and in control compared to just talking to the robots through chat. This system shows new ways to better supervise and adjust robot teams using language.
What this means in practice
- •For robotics operations teams: Manage and intervene in heterogeneous robot teams using a mixed reality interface that clarifies plans and execution with language-based commands.
- •For industrial automation managers: Improve situational awareness and control over multi-robot workflows by supervising robots in immersive environments linked to language-based coordination.
Authors
Xiaoran Yang, Xun Qian, Yang Zhan, Nathan Tran, Ziyi Liu, Qiao Jin
Abstract
Large language models (LLMs) let users direct heterogeneous multi-robot systems (MRS) through natural language, but make task interpretation, robot assignment, and coordination difficult to inspect and change. Based on a formative study with 12 non-expert users, we developed MRPilot, a mixed reality system organized around four stages of supervision and intervention. MRPilot represents robot-team plans and execution states as structured commitments shared across synchronized situated and overview views. Across four stages, it helps users resolve ambiguous references (Forming), review plans before execution (Reviewing), monitor distributed execution (Following), and make robot-level or team-level changes when problems arise (Repairing). In a within-subjects study with 20 participants in a virtual reality-simulated home, MRPilot reduced workload, increased situational awareness, transparency, trust, and perceived control compared with a conventional LLM-based conversational interface using the same LLM planner and robot capabilities. We provide design implications for multi-scale intervention, adaptive supervision, and calibrated reliance in LLM-based MRS.