Speech dubbing method keeps translated audio in sync with video
X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints
Computation and LanguageComputer Vision and Pattern Recognition
Summary
When dubbing videos from one language to another, it’s hard to keep the translated speech matching the timing and mouth movements of the original. The authors present X-Dubbing, a method that translates speech into text and then back to speech while respecting timing and visual cues from the video. This approach trains two parts separately without needing special paired dubbing examples. Their experiments show it does better than previous systems at keeping dubbed speech synchronized and looking natural with the video.
What this means in practice
- •For film dubbing studios: Produce dubbed videos where translated speech matches actors’ lip movements and timing more naturally without needing extensive paired dubbing recordings.$Commercial implications: Enables selling advanced dubbing software that improves synchronization quality and reduces production costs.
- •For video conferencing platforms: Automatically translate and dub live or recorded multilingual video calls with speech closely aligned to the speaker’s video, enhancing viewer experience.
Authors
Jeongsoo Choi, Ji-Hoon Kim, Junseok Ahn, Joon Son Chung
Abstract
Cross-lingual dubbing aims to convert source speech into target-language speech that remains synchronized with the original video. The key to this task is to produce a faithful translation that is naturally speakable within the source timing, which is non-trivial because languages differ in duration, rhythm, and articulation. In this paper, we propose X-Dubbing, which formulates the task as a joint problem of time-aware speech-to-text translation and video-aware text-to-speech. Specifically, our method consists of a Dubbing Translator that generates temporally structured scripts and a Voice Actor that renders them in synchrony with the source video, both grounded in a language-agnostic timing budget. We further refine the translator with dubbing-aware feedback from the synthesized speech, encouraging scripts that yield well-synchronized dubbing. This design requires no paired dubbing data, as the two components are trained on speech-text and monolingual audio-visual data respectively. Extensive experiments show that X-Dubbing outperforms recent baselines across diverse metrics, including isochrony and audio-visual correspondence.