Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization
2026-06-09 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed Lip Forcing, a new method to sync lip movements in videos with audio quickly and accurately. They created smaller, faster models by learning from a large, slow but precise teacher model, allowing real-time lip syncing with fewer processing steps. Their approach balances keeping video quality and syncing lips well using special techniques based on analyzing how sync quality changes. They tested two versions of their model, showing much faster speeds without losing quality compared to previous methods.
lip synchronizationdiffusion modelsautoregressive modelsvideo-to-video (V2V)denoising stepsbidirectional attentioncausal modelsClassifier-Free Guidance (CFG)real-time inferenceknowledge distillation
Authors
Paul Hyunbin Cho, Jinhyuk Jang, SeokYoung Lee, Joungbin Lee, Siyoon Jin, Heeseong Shin, Jung Yi, Yunjin Park, Chulmin Park, Seungryong Kim
Abstract
Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, $17.6\times$ faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs $39.8\times$ faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.