An Augmented Reality Brain-Robot Interface for Generalist Robot Arm Manipulation
2026-06-15 • Robotics
RoboticsHuman-Computer Interaction
AI summaryⓘ
The authors created a system that lets people control a robot arm using both eye movements and brain signals. They combined eye tracking to choose objects and brain-based commands to move the robot, helping users perform everyday tasks like drinking or opening a drawer. They tested it with 18 healthy participants and found that it was easy to use and kept people engaged. Their results show that this approach could help people control robots in more flexible ways, encouraging further tests with those who might really need this technology.
augmented realitybrain-computer interfaceEEGeye trackingmotor imageryrobot arm manipulationshared autonomyusability studyassistive roboticssequential task execution
Authors
Shangkai Zhang, Rousslan Fernand Julien Dossa, Luca Nunziante, Marina Di Vincenzo, Kai Arulkumaran
Abstract
The integration of augmented reality (AR) and EEG-based brain-computer interfaces (BCIs) offers a promising path for enabling intuitive control of robots for assistive purposes. However, existing AR brain-robot interface (BRI) systems are often constrained to task-specific structures, limiting their utility in real-world environments. We present an AR BRI designed for generalist robot arm manipulation that combines gaze-based object selection with motor imagery action control. Our system uses eye-tracking for intuitive object targeting and context-aware visual overlays ("Place" and "Use") to guide the user through tasks within a shared autonomy framework. We evaluated the interface through a feasibility study with 18 healthy participants performing three multi-step activities of daily living: drinking, using a drawer, and operating an oven. Our results demonstrate that this interaction paradigm enables effective sequential task execution and high user engagement, achieving a "Good" usability rating (SUS > 70). These findings support the feasibility of the proposed interaction paradigm for complex BCI-driven robotic assistance, and motivate future evaluation with the intended target population. Project website: https://ar-bri-manip.github.io/.