Audio lip-sync method recreates person specific mouth details
Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation
Computer Vision and Pattern Recognition
Summary
Most current systems that animate talking faces produce generic mouths that do not match the real person's unique lip and teeth details. The authors propose RGOR, which improves this by using reference images of the person’s own mouth to guide the animation. RGOR is trained with a special judge model that checks if the generated mouth matches the person's real mouth, not just any plausible mouth. Tests show RGOR keeps the person's mouth details while syncing talking animations accurately.
What this means in practice
- •For film post-production teams: Create more realistic dubbed face animations that preserve each actor’s unique mouth features for better audience immersion.
- •For virtual avatar developers: Generate audio-driven talking avatars that maintain the original person’s lip and dental details for personalized digital characters.$Commercial implications: Enables selling personalized avatar creation software that requires matching the user’s unique mouth features precisely.
Authors
Bangxun Tang
Abstract
We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.