Augmented reality system streams robot sensor data for better collaboration
Augmented Reality Interfaces for Human-Robot Collaboration: Development of a ROS 2-Based Sensor Streaming Framework and Validation via SLAM Algorithms
Robotics
Summary
Working safely and effectively with robots in factories means humans and robots need to understand each other well. This paper describes a new way to connect an augmented reality headset with a popular robot software system to share real-time sensor data. The authors built a program that streams the headset's position and camera data to robot software and tested it using methods that map spaces and track movement. Their tests show the system works reliably and quickly, which could help humans and robots work together more smoothly and safely.
What this means in practice
- •For industrial automation engineers: Integrate augmented reality headsets with robot control systems to improve spatial awareness and interaction safety on factory floors.
- •For augmented reality developers: Create real-time AR applications that share sensor and environmental data with robotic platforms for enhanced mixed-reality collaboration.
Authors
Alessandro Rubert, Stefano Ghidoni, Matteo Terreran
Abstract
In recent years, Human-Robot Collaboration (HRC) has taken on a central role in Industry 4.0 and collaborative robotics, demanding communication channels that are increasingly bidirectional, intuitive, and efficient. In this context, Augmented Reality (AR) presents itself as a fundamental enabling technology, capable of both displaying information to the operator and gathering spatial data about the surrounding environment. This thesis presents the development of a sensor streaming framework that connects the Magic Leap 2 AR headset with the ROS 2 (Robot Operating System) ecosystem. Using the Unity development environment and the ROSTCP-Connector package, an on-board application for the headset was developed, capable of acquiring real-time data from the integrated sensors (pose tracking, cameras, and environmental sensors) and publishing it to dedicated ROS 2 topics. In order to test the accuracy, latency, and robustness of the generated data stream, the framework was validated using SLAM (Simultaneous Localization and Mapping) algorithms known in the literature. The experimental results demonstrate that the proposed architecture ensures stable data transmission, laying the groundwork for safe real-time interaction and shared spatial awareness, and opening up new perspectives for the control and supervision of robotic systems in complex HRC scenarios.