Robots personalize gestures to match users without retraining models
Co-Speech with You: Training-Free Personalization of Robot Co-Speech Gestures
Robotics
Summary
Getting robots to mimic a person's unique hand and body movements while they talk usually needs retraining the robot's software, which takes time and effort. The authors developed a way for robots to learn a new person's gesture style quickly and without retraining, by using about 10 seconds of that person's motion recordings. They tested their method and found it can recognize and reproduce personalized gestures well, even when the robot talks with different synthetic voices. This helps robots feel more natural and connected during conversations.
What this means in practice
- •For robotics developers: Enable social robots to quickly adopt a user's gesturing style without retraining, improving interaction naturalness in diverse user settings.
- •For virtual avatar creators: Create personalized co-speech gestures for avatars from brief recordings, enhancing realism in interactive virtual agents and games.
Authors
Bosong Ding, Selma Ancel, Giacomo Spigler, Murat Kirtay
Abstract
Personal robots should adapt their co-speech gesture style to a new user without requiring model retraining. We present a training-free personalization pipeline that combines a frozen audio-conditioned diffusion prior with a gesture style encoder and lightweight conditioning adapters. The encoder is first trained to discriminate speaker identities and then jointly refined with the adapters using the diffusion objective, enabling a reusable style embedding to be extracted from approximately 10 seconds of enrollment motion through a single forward pass. To support this setting, we also release a Quest~3 capture application and a dataset of spontaneous co-speech motion from ten participants. We evaluate the system on held-out speakers using Style Recognition Accuracy (SRA) and Fr'echet Gesture Distance (FGD) to measure personalization and motion quality. Our approach improves SRA from 27.6\% for the frozen prior to 69.5\% while preserving motion quality (FGD 34.2 versus 34.8), and replacing the enrollment embedding with another person's reduces SRA to 11.4\%. The generated gestures are retargeted to a physical NAO robot, and this improvement also transfers to the robot deployment setting, where speech is synthesized using five TTS voices, retaining 67.3\% SRA.