Composing audio and text to create synchronized whole-body gestures

CFGComposer: Compositional Audio-Text-to-Motion Generation via Copula Score Correction

SoundArtificial Intelligence

Summary

Making robots or digital characters move their bodies in ways that match both what they are hearing and what they are supposed to be expressing is very hard. The authors create a new approach that combines two separate motion generators—one that listens to audio beats and one that understands text meaning—and corrects their combined output to better capture interactions between sound and meaning. Their approach produces more accurate and natural-looking gestures that align well with speech rhythm and text meaning. They tested it on a standard dataset and even showed it working on a real robot.

What this means in practice

  • For social robotics developers: Generate whole-body gestures for robots that are rhythmically synced with speech and semantically matched to spoken content for more natural interactions.
  • For animation and game studios: Create synchronized character animations that correspond to both speech rhythm and descriptive text without manual keyframe work.
  • For humanoid robot manufacturers: Deploy gesture generation models directly on robots for real-time, natural motion driven by speech audio and text scripts.$Commercial implications: This enables selling advanced humanoid robots with natural gesturing abilities based on synchronized audio-text inputs.

Authors

Pengyang Shi, Jinlu Zhang, Chunze Lin, Yizhou Wang, Li Yi

Abstract

Generating whole-body gestures that are simultaneously rhythmically synchronized with speech audio and semantically aligned with text descriptions is a fundamental challenge in embodied AI and social robotics. We tackle this Audio-Text-to-Motion (AT2M) problem by composing two independently-trained diffusion models at inference time and refine bias in joint inference by modeling the copula score $s_I$. A rigorous theoretical analysis reveals that naive epsilon blending corresponds to a Product-of-Experts formulation that omits the copula score $s_I$: the cross-modal interaction between audio and text that neither marginal model can capture. We propose CFGComposer: a jointly-trained AT2M diffusion model that estimates $s_I$ via a CFG analogy and injects it as an $ε$-space correction. Experiments on BEAT demonstrate best-in-class semantic fidelity (SRGR $0.9916$, FID $27.18$) with GT-level beat alignment (BA $0.1826$ vs. GT $0.1743$), and 76-83% win rate in blind pairwise human evaluation against all prior methods. Physical deployment on a Unitree G1 humanoid confirms direct sim-to-real transfer.