Avatar motion generated naturally from speech text and path inputs
Multi-Modal Controlled Coherent Motion Generation
Computer Vision and Pattern Recognition
Summary
People often walk and talk at the same time naturally, but making 3D avatars do this realistically is hard. The authors developed a new method that can mix different types of input—like speech sounds, text descriptions, and movement paths—to create smooth, lifelike avatar motions. Their approach builds the motion step-by-step, letting each input contribute separately before combining them in a way that looks natural. This method works well even when the inputs aren't perfectly matched, producing more realistic movements than earlier techniques.
What this means in practice
- •For game developers: Create 3D characters that move naturally by combining player voice, commands, and navigation paths into coherent animations.$Commercial implications: Enables game studios to produce more immersive characters by synchronizing motion with speech and text inputs in real time.
- •For virtual event producers: Generate realistic avatar motions that combine scripted dialogue and live audio to improve social interaction in virtual events.
Authors
Yifei Liu, Qiong Cao, Hongwei Yi, Huaiguang Jiang, Changxing Ding
Abstract
It is natural for humans to walk and talk simultaneously. This paper tackles the challenge of replicating such natural behaviors in 3D avatar motion generation driven by concurrent multimodal inputs, such as a text description of a man walking alongside speech audio. Existing methods, constrained by the scarcity of aligned multimodal data, typically combine motions from individual modalities sequentially or through weighted sums. However, they often result in mismatched or unrealistic movements. To overcome these limitations, we propose MOCO, a novel diffusion-based framework capable of processing multiple simultaneous inputs, including speech audio, text descriptions, and trajectory data, to generate coherent and lifelike motions without requiring aligned multimodal data. Our key innovation lies in decoupling the motion generation process. During each denoising step, the diffusion model independently generates motions for each modality from the input noise and assembles the body parts according to predefined spatial rules. The resulting combined motion is then diffused and serves as the input noise for the subsequent denoising step. This iterative approach enables each modality to refine its contribution within the context of the overall motion, progressively harmonizing movements across modalities. Consequently, the generated motions become increasingly natural and fluid with each iteration, achieving coherent and synchronized behaviors. We evaluate our approach using a purpose-built multimodal benchmark. Experimental results demonstrate that MOCO outperforms existing baselines, advancing the field of multimodal motion generation for 3D avatars.