VidDub improves video dubbing by syncing lip movement to new speech
VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing
Computer Vision and Pattern Recognition
Summary
Video dubbing changes the lip movements in a video to match new spoken words while keeping the person’s identity and background the same. This is hard because the original mouth shapes conflict with the new speech. The authors introduced a way to ignore the old mouth shapes during training and instead focus on matching the lips to the new audio. They built a system called VidDub that does this efficiently in just two steps and tested it on several benchmarks, showing better lip syncing and overall video quality.
What this means in practice
- •For video editors: Edit videos to match new voiceovers with accurate lip movements and preserved face and background details.
- •For content localization teams: Produce dubbed videos that visually align lip motions with translated speech for better audience immersion.
Authors
Yexin Liu, Yunlong Lin, Zehan Wang, Guangxin He, Xinli Xu, Jiajun Zha, Yaofu Liu, Yueze Wang, Bowen Ping, Yuanming Li, Amie Wang, Bernard Ghanem, Bing Li, Harry Yang
Abstract
Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced. We introduce replacement-aware cross-audio supervision: paired-audio samples use full-frame reconstruction, whereas unpaired replacement-audio samples mask out source-mouth reconstruction losses and supervise the mouth solely through audio-visual synchronization while preserving unmasked facial regions and background. Because offline dubbing provides the complete source video and target speech beforehand, we present VidDub, a full-context diffusion editor distilled into two denoising steps (2 NFE). Our replacement-aware teacher first learns accurate lip synchronization without regressing toward conflicting source mouth movements. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student, and distribution matching distillation recovers visual quality after step compression. We will release the curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Across three training seeds, masking out conflicting source-mouth reconstruction improves cross-audio Sync-C by 0.833; the final two-step model combines strong lip synchronization with superior visual quality, identity preservation, and visual speech recognition across four benchmarks.