Voice activity signals improve speech timing for better lip-synced dubbing
Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing
Artificial Intelligence
Summary
Matching the timing of speech and lip movements is important for making dubbed videos look natural. The authors propose using a simple voice activity signal—basically marking when speech is on or off—to guide speech synthesis so that it matches the original video's timing more closely. Their method keeps the speech sounding natural with proper pauses and rhythm. It also allows editors to choose how strictly the speech should follow the timing during dubbing.
What this means in practice
- •For video production teams: Generate dubbed speech that closely matches original lip movements using voice activity cues for better viewer experience.$Commercial implications: Enables commercial dubbing tools to create more natural lip-synced speech in multiple languages, improving localized video quality.
- •For speech technology developers: Incorporate voice-activity signals into speech synthesis systems to better control speech timing and pauses in generated audio.
Authors
Alejandro Pérez-González-de-Martos, Florian Lux, Angelina Elizarova, Milana Shkhanukova, Andreas Kellner, Mattia Antonino Di Gangi
Abstract
Automatic lip-synchronous dubbing requires a speech synthesis model to generate alternating voice and silence patterns in the target language that match the timing of the source clip precisely to ensure an optimal viewing experience. Prior works address this problem by conditioning the speech synthesis process on lip movements extracted from the video signal. In this work, we condition the speech generation on a binary voice-activity signal, which has a lightweight representation and can be produced in multiple ways. We show that the model follows the voice-activity signal with high accuracy while maintaining natural prosody and semantically appropriate pause placement within sentences, as demonstrated through extensive objective and subjective evaluations. By randomly masking this condition during training, we make the feature entirely optional during inference, allowing editors to enforce or relax lip-sync constraints when desired.