MIRA enables real-time interactive motions for social robots
MIRA: Real-Time Full-Duplex Human-Robot Interaction for Embodied Companions
Robotics
Summary
Interactive robots need to understand what a person wants, respond quickly, and move naturally at the same time. The authors created MIRA, a system that listens to streaming speech, predicts what the robot should say next, and generates matching gestures that can be changed quickly if needed. Their approach keeps the robot’s movements safe and smooth while talking with people. They tested MIRA on a humanoid robot and found it can respond promptly and handle interruptions better than previous methods.
What this means in practice
- •For robotics developers: Build social robots that can talk and move naturally together while responding quickly to user speech in real time.
- •For interactive entertainment teams: Create humanoid robots for live events that synchronize speech and gestures fluidly, improving audience engagement through responsive behavior.
Authors
Lijian Lin, Ye Zhu, Fan Zhang, Yunfei Liu, Baofeng Li, Xianwen Zeng, Jianan Wang, Yu Li
Abstract
Real-time embodied companion interaction requires a robot to infer user intent from streaming speech, generate timely responses, and execute expressive, interruptible motions. Existing systems typically decouple dialogue orchestration from gesture synthesis, relying on offline motion generation from complete audio. This separation leaves open how a deployed robot can dynamically synchronize response content, prosodic timing, and physical safety under incremental inputs and uncertain turn boundaries. We present MIRA, a unified framework for full-duplex embodied companion interaction. Given streaming user speech, dialogue history, and vocal affect, MIRA predicts both the response text and an explicit embodiment cue. Discrete social behaviors (\eg listening, greeting) are mapped to validated robot trajectories, while open-ended speaking is paired with streaming, co-speech motion. This generative motion is governed by a predict-more-than-commit sliding window that provides temporal look-ahead for motion continuity while limiting physical commitment to a short, cancellable prefix. Crucially, we design CORTEX, a dual-timescale interaction policy that manages low-latency streaming and deliberative turn decisions, backed by a robot-side execution layer that enforces physical safety constraints at the control rate. We deploy MIRA on an Astribot S1 humanoid robot. Quantitative evaluations demonstrate competitive audio-motion alignment relative to state-of-the-art motion-generation baselines, while real-robot deployment measurements characterize streaming responsiveness and interruption handling.