Papers for

film dubbing teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Real-time lip syncing with emotion control for all character styles

Emo-LiveSync: Real-Time Universal Lip Synchronization with Explicit Emotion Control

Abstract: Lip synchronization, the task of generating realistic lip movements synchronized with audio signals, is fundamental to applications such as film dubbing, virtual avatars, and digital content creation. Recent GAN-based methods offer computational efficiency but often lack visual fidelity, while diffusion-based approaches achieve high-quality results at the cost of substantial computational overhead. Moreover, most existing methods are limited to photo-realistic human faces, restricting their applicability to stylized characters and non-human entities. In this paper, we propose an efficient universal lip-sync video generation framework that achieves high-quality results without compromising computational efficiency. We build on LivePortrait, a warp-based portrait animation model, and introduce a training strategy that aligns audio signals to an implicit motion space by distilling knowledge from a pre-trained video-to-motion model. This enables real-time inference while maintaining high visual fidelity, and naturally generalizes to diverse stylistic characters beyond photo-realistic humans. Furthermore, we introduce an expressive control gate that enables explicit manipulation of emotional expressions for content enrichment. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both visual quality and lip-sync accuracy while ensuring real-time inference.

Fri 6 NovComputer Vision and Pattern Recognition
The gist
Making characters move their lips so they match what they are saying is tricky, especially if you want it to look good and work fast. The authors improved a way to do this by teaching a computer to understand the sounds and turn them into natural lip movements in real time. Their method works not just for realistic human faces but also for cartoon or stylized characters, and it lets you control the character’s emotions as they speak. This makes creating videos and animations with talking characters easier and more expressive.
Open → 2611.09074v1