Speech encoder improves lip movement accuracy in 3d facial animation
CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation
Computer Vision and Pattern RecognitionSound
Summary
Animating realistic lip and facial movements from speech sounds is challenging because current speech analysis tools don’t focus on the visible mouth shapes needed. The authors designed a new speech encoder called CASE that better captures how lips move by learning from 3D models of lip shapes. This improved encoder combines with existing speech models to make facial animations more accurate at matching spoken sounds. Their tests showed that CASE reduces errors in lip and face shape prediction across multiple animation systems, even when trained on Korean speech and tested on English.
What this means in practice
- •For animation studios: Create more accurate lip-sync and facial animations driven directly by speech audio inputs.$Commercial implications: Enables studios to produce higher quality speech-driven 3D facial animations for films and games with improved naturalness and lower manual correction costs.
- •For game developers: Enhance character expressions in real-time gameplay by integrating speech-driven facial animation with improved lip movement accuracy.
Authors
Jaehong Myung, Seungho Eum, Taegon Kim, Hyunjung Chung, Unsang Park
Abstract
General-purpose self-supervised speech encoders provide strong acoustic representations, but their pretraining objectives are not explicitly aligned with the visible articulatory dynamics required by speech-driven 3D facial animation. We propose CASE (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents. After adaptation, the geometry branch is removed, and the audio-only CASE representation is fused with the baseline SSL speech representation through residual cross-attention, while retaining the motion-decoder architecture and objective. Across four VOCASET baselines, CASE reduces mean lip vertex error by 2.8-6.0%, and mean full-face vertex error decreases in every system. Under the same fusion protocol, CASE yields lower mean LVE than the unadapted AV-SE stream on all four generators, with model-dependent margins. Because adaptation uses Korean speech-geometry pairs and evaluation uses English VOCASET, the result is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.