Social robots combine speech vision and touch in virtual reality
Multimodal Socially Aware Framework for Human-Robot Interaction in Virtual Reality
Robotics
Summary
It is hard for robots to interact naturally with people because they need to understand speech, sight, and touch all at once. The authors created a system that helps robots do this in virtual reality by coordinating their words and movements based on what they see, hear, and feel. When tested with people, the system was somewhat natural but not as good as a human controlling the robot directly. The study shows that timing, clarifying unclear requests, and space coordination are big challenges for autonomous social robots.
What this means in practice
- •For vr developers: Build virtual reality experiences where social robots respond naturally to user speech, vision, and touch inputs.
- •For robotics engineers: Design autonomous social robots that coordinate multimodal inputs to produce context-appropriate speech and movement.
Authors
Richard Xu, Dakarai Crowder, Alexis E. Block, Wenzhen Yuan
Abstract
Social robots must coordinate speech and movement while interpreting multimodal human input. We present a multimodal framework that integrates speech, vision, and touch to generate contextually appropriate robot speech and movement. We evaluated the framework in virtual reality with 18 participants across five social interaction scenarios, comparing its performance against a human-teleoperated baseline. Although participants rated the autonomous system above the neutral midpoint for both naturalness and social appropriateness, the teleoperated baseline received significantly higher ratings on both measures. Behavioral coding and qualitative feedback revealed that conversational timing, clarification of ambiguous requests, and spatial coordination were important factors contributing to this performance gap. These findings highlight key challenges in developing autonomous robots capable of natural, multimodal social interaction.