Streaming talking-head videos generated efficiently with improved quality
Decoupled Self-Forcing Distillation for Streaming Talking Head Generation
Computer Vision and Pattern Recognition
Summary
Making videos of talking heads in real time is tricky because methods that focus on quality usually need very big models and are slow, while faster methods don’t look as good. The authors found that mixing audio with the entire video information wastes effort because the audio mostly affects motion, not appearance or background. Their solution is to separate motion from identity and appearance, using a smaller model to predict motion from audio and a separate model to create video frames from motion. With a special training method balancing these models, they can generate high-quality streaming videos quickly and efficiently.
What this means in practice
- •For video communication developers: Create real-time talking-head avatars from audio input with reduced delay and better visual quality by decoupling motion from facial identity.
- •For interactive media producers: Build interactive avatar systems that generate convincing facial movements on the fly without needing extensive computational resources.
Authors
Yanru An, Ruiyan Wang, Wenwu Wei, Rui Bu, Qi Wang, Hongwei Hu, Zhengxue Cheng, Rong Xie, Li Song, Wenjun Zhang
Abstract
Streaming talking-head generation produces each frame as its driving audio arrives, yet fidelity and efficiency have so far pulled in opposite directions: end-to-end methods condition a video diffusion model on audio directly and achieve high quality but only at large scale, while cheaper two-stage methods generate an intermediate motion representation and trail in fidelity. We argue the cost of the former lies in the target of fusion: the video latent is dominated by identity, appearance and background, none of which audio bears on, so coupling audio to every pixel blurs detail and wastes capacity. We instead fuse conditions in a low-dimensional identity-disentangled motion space, routing audio and motion captions by their temporal granularity, and generate motion latents with a small causal autoregressive transformer that a pretrained diffusion renderer turns into video. Conditions thus control video transitively, and high fidelity no longer requires a large backbone. Streaming this decomposition needs both models to be causal, and the exposure-bias problem could be solved by self-forcing given a bidirectional teacher. But there is no such teacher in motion space. Our decoupled self-forcing distillation resolves both models under one frozen teacher: conditioned on motion, it distills the renderer into a block-causal student; unconditionally, it scores rendered rollouts against real videos, supervising motion by the video it produces. This lifts the fidelity ceiling from the motion generator onto the stronger renderer. The two models run as parallel causal streams, reaching 15.4 FPS at 1.3 s latency with no quality degradation.