Vibeavatar improves realistic talking avatar lip sync and motion quality
VibeAvatar: Aligning Phonetic Kinematics and Human Aesthetics for High-Fidelity Talking Avatar Synthesis
Computer Vision and Pattern Recognition
Summary
Making talking avatars that look and move realistically is hard because getting both accurate lip movements and natural-looking facial motions at the same time is tricky. The authors of this paper found that these two goals come from different needs and should be handled separately. They created VibeAvatar, which uses one part to focus on making lips move correctly with speech sounds and another part to make the overall motion look good and natural. This approach leads to better-looking, more accurate talking avatars that can be made efficiently on a computer.
What this means in practice
- •For video game developers: Generate realistic talking characters with accurate lip sync and natural facial motion for immersive gaming experiences.$Commercial implications: Enables creation of high-quality talking avatars that can be sold as part of interactive games enhancing user engagement.
- •For animation studios: Produce high-fidelity talking avatars for animated films or virtual actors with improved lip movement and motion aesthetics.$Commercial implications: Supports production of more lifelike animation content, which can be monetized as movies or digital media products.
Authors
Qilin Wang, Mingyu Li, Hao Tang
Abstract
Multi-modal talking avatar synthesis aims to generate realistic talking videos from a reference portrait and speech. Despite rapid progress in diffusion-based methods, existing approaches still struggle to jointly achieve accurate lip articulation, human-preferred motion aesthetics, and efficient inference. We observe that phonetic accuracy and motion aesthetics arise from fundamentally different sources and should be addressed at complementary stages rather than learned implicitly by a single generator. Based on this insight, we propose VibeAvatar, which disentangles these two objectives through a Phonetic Kinematics Adapter (PKA) that converts recognition-oriented speech features into phonetic-kinematic conditions at the conditioning stage, and an Aesthetic Motion Policy (AMP) that optimizes a flow-consistent stochastic sampling policy via Group Relative Policy Optimization (GRPO) at the post-training stage. With a lightweight flow-based motion generator operating in a compact 1D warp-based latent motion space, VibeAvatar achieves state-of-the-art results in articulation, aesthetics, and efficiency on both objective metrics and user studies, while generating a 10-second 512px video in under 10 seconds with only $\sim$3GB VRAM.