MegaAvatar generates talking avatars with full body and face control
MegaAvatar: Controllable Talking Avatar Generation
Computer Vision and Pattern Recognition
Summary
Creating digital avatars that can talk and move realistically is hard because you need to control both the face and body motions. This paper shows MegaAvatar, a new system that uses 3D body models along with audio and face information to animate avatars more naturally. It can produce avatars that speak with matching facial expressions and body movements, and the avatars keep looking like the original person. The system can work flexibly with different video lengths and resolutions.
What this means in practice
- •For game developers: Generate realistic talking characters with controllable full-body motion and synchronized speech expressions for immersive gaming experiences.$Commercial implications: Enables creation of high-quality avatars for games that require lip sync and body animation, enhancing player engagement and realism.
- •For video content creators: Create personalized talking avatars that match a speaker’s voice and motions to use in videos without extensive motion capture equipment.
Authors
Junyao Gao, Sibo Liu, Weidong Zhang, Cairong Zhao, Jun Zhang
Abstract
This report presents \textbf{MegaAvatar}, a controllable talking avatar generation framework built on top of the Wan2.2-TI2V-5B model. Compared with previous talking-avatar methods that mainly rely on audio or reference-image conditioning, we introduce additional SMPL-X-derived 3D guidance, enabling global control over body pose and head motion. Specifically, we render the driving SMPL-X sequence into dense mesh frames and encode them with a lightweight 3D convolutional encoder, whose outputs are injected into the latent tokens to provide overall motion control. Furthermore, we extend Wan2.2-TI2V-5B with additional audio and face cross-attention modules to enable fine-grained expression control and preserve the input identity, respectively. In addition, we implement an audio-to-SMPL-X model to predict an SMPL-X sequence conditioned on the reference image and input audio, allowing MegaAvatar to support audio-driven inference without user-provided SMPL-X frames. Experiments show that MegaAvatar achieves high-quality talking avatar generation with controllable body and head motion, speech-synchronized facial expressions, and consistent identity preservation. MegaAvatar also supports inference with flexible resolutions and video lengths. Codes, dataset, models will be avaliable in https://github.com/Jeoyal/MegaAvatar