Wavelet diffusion improves realistic long speech driven 3d motion
DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion
Graphics
Summary
Animating realistic 3D body and facial movements that match speech is difficult because different kinds of motion happen at different speeds and details can get lost. The authors created a system called DynaConTalk that separates motion into layers, like slow body movements and quick hand gestures, using a mathematical tool called wavelet transform. Their method lets the animation follow speech rhythms and words more accurately, and it can also be controlled and corrected easily. This results in more natural and detailed animations that keep timing and facial expressions consistent over a long time.
What this means in practice
- •For animation studios: Create long-form, speech-driven 3D character animations with improved natural motion and facial expression syncing using DynaConTalk's wavelet-based diffusion approach.$Commercial implications: Enables high-quality, controllable virtual character animation for films, games, and AR/VR products, differentiating offerings with realistic speech-driven motion.
- •For virtual assistant developers: Generate detailed and temporally consistent 3D gestures and facial expressions for avatars that align closely with spoken content in real time or playback.
Authors
Yifei Zhu, Yangyang Cai, Mingyi Shi, Miao Cheng, Lin Gu, Taku Komura, Yoshifumi Kitamura
Abstract
Holistic co-speech animation is prone to averaging in both motion representation and speech conditioning. In coordinate-space diffusion, slow body posture, mid-frequency gesture strokes, and fast hand or facial details are entangled in one prediction target, often producing low-variance, over-smoothed motion. Meanwhile, dense rhythmic and acoustic cues can dominate sparse content-specific information under fixed multimodal fusion. We present DynaConTalk, a wavelet-constrained diffusion framework for long-form and controllable holistic co-speech motion generation. Diffusion operates in stationary wavelet transform (SWT) coefficient space, whose temporally aligned bands separate coarse posture evolution, gesture strokes, and fine expressive details. Our dynamic gating network preserves HuBERT and speaker identity as a base and selectively adds rhythm, mel, and transcript features through motion-state- and noise-aware residual gates. Attention pooling and learned depth routing deliver complementary conditions to each denoising stage, while a frame-resolution rhythm path preserves precise timing. A signed proposal-consensus update then reconciles these conditions with the evolving motion state. Matched-noise constraint injection uses the same sampling interface for history continuation and localized keypose repair, and extends to reference-guided control. Separate body-hand and facial denoisers, followed by inverse SWT and a pose-driven root regressor, produce holistic motion. Experiments evaluate generation quality, facial accuracy, temporal continuity, and controllable editing. Code, models, and the interactive editing interface are available at https://github.com/zhuyifeiabcd1/DynaConTalk.