Socialhumanoid generates expressive full-body robot motions with speech
SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation
RoboticsComputer Vision and Pattern Recognition
Summary
Humanoid robots need to move naturally and expressively while talking with people, but making their movements match speech in real time is hard. The researchers created SocialHumanoid, a system that generates body motions synchronized with speech and emotional cues in a single step, enabling smooth and fast robot movements. They also introduced a new dataset of actors showing body motions with emotions to help the system learn. Their system works better and faster than some previous methods and can run on real robots continuously.
What this means in practice
- •For robot developers: Generate real-time expressive body motions that match speech and emotion for humanoid robots during face-to-face interaction.
- •For virtual character designers: Create natural co-speech full-body animations conditioned on affective states to improve character expressiveness and synchrony.
Authors
Chengqun Yang, Tengjie Zhu, Liang Xu, Fulong Liu, Guanzhu Ren, Yitong Xing, Xuefeng Lu, Fei Shi, Siyuan Fan, Weijie Dong, Yao Mu, Xiaokang Yang, Yichao Yan
Abstract
Humanoid robots are increasingly expected to serve as embodied social agents that communicate naturally with humans through face-to-face interaction. During such communication, humanoid robots require body behaviors that are synchronized with speech, affectively expressive, and suitable for real-time execution. However, existing co-speech methods are primarily developed for digital humans and lack joint support for affective control and low-latency continuous generation on physical embodiments. To bridge this gap, we present SocialHumanoid, a system for expressive humanoid behavior via one-step co-speech motion generation. Given response speech and a specified affective condition, SocialHumanoid generates each full-body motion window in a single forward pass and connects successive windows through motion-history conditioning. The generated human motion is further converted online into embodiment-compatible robot references and tracked by a whole-body controller for physical execution. To provide explicit supervision for affective body expression, we further introduce AffectMoCap, a 4-hour dataset captured from two professional actors, containing synchronized speech, body motion, fine-grained hand motion, and emotion annotations. On BEAT2, SocialHumanoid achieves the best FGD among the compared generation methods, competitive speech-motion synchrony, and approximately $6\times$ faster inference than GestureLSM under the same protocol. Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution. Our project page is https://rex0191.github.io/SocialHumanoid/.