Real-time lip syncing with emotion control for all character styles

Emo-LiveSync: Real-Time Universal Lip Synchronization with Explicit Emotion Control

Computer Vision and Pattern Recognition

Summary

Making characters move their lips so they match what they are saying is tricky, especially if you want it to look good and work fast. The authors improved a way to do this by teaching a computer to understand the sounds and turn them into natural lip movements in real time. Their method works not just for realistic human faces but also for cartoon or stylized characters, and it lets you control the character’s emotions as they speak. This makes creating videos and animations with talking characters easier and more expressive.

What this means in practice

  • For film dubbing teams: Generate lip-synced video with emotion adjustments for diverse characters instantly to improve dubbing quality.$Commercial implications: Enables movies and shows to sell dubbed versions with better visual coherence and emotional expression.
  • For virtual avatar developers: Create expressive talking avatars in real time that reflect speech and emotions across different art styles.

Authors

Sangjoon Yu, Seungeun Lee, JoonHeyok Shin, Ji-Su Kang

Abstract

Lip synchronization, the task of generating realistic lip movements synchronized with audio signals, is fundamental to applications such as film dubbing, virtual avatars, and digital content creation. Recent GAN-based methods offer computational efficiency but often lack visual fidelity, while diffusion-based approaches achieve high-quality results at the cost of substantial computational overhead. Moreover, most existing methods are limited to photo-realistic human faces, restricting their applicability to stylized characters and non-human entities. In this paper, we propose an efficient universal lip-sync video generation framework that achieves high-quality results without compromising computational efficiency. We build on LivePortrait, a warp-based portrait animation model, and introduce a training strategy that aligns audio signals to an implicit motion space by distilling knowledge from a pre-trained video-to-motion model. This enables real-time inference while maintaining high visual fidelity, and naturally generalizes to diverse stylistic characters beyond photo-realistic humans. Furthermore, we introduce an expressive control gate that enables explicit manipulation of emotional expressions for content enrichment. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both visual quality and lip-sync accuracy while ensuring real-time inference.