Papers for

video communication developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Streaming talking-head videos generated efficiently with improved quality

Decoupled Self-Forcing Distillation for Streaming Talking Head Generation

Abstract: Streaming talking-head generation produces each frame as its driving audio arrives, yet fidelity and efficiency have so far pulled in opposite directions: end-to-end methods condition a video diffusion model on audio directly and achieve high quality but only at large scale, while cheaper two-stage methods generate an intermediate motion representation and trail in fidelity. We argue the cost of the former lies in the target of fusion: the video latent is dominated by identity, appearance and background, none of which audio bears on, so coupling audio to every pixel blurs detail and wastes capacity. We instead fuse conditions in a low-dimensional identity-disentangled motion space, routing audio and motion captions by their temporal granularity, and generate motion latents with a small causal autoregressive transformer that a pretrained diffusion renderer turns into video. Conditions thus control video transitively, and high fidelity no longer requires a large backbone. Streaming this decomposition needs both models to be causal, and the exposure-bias problem could be solved by self-forcing given a bidirectional teacher. But there is no such teacher in motion space. Our decoupled self-forcing distillation resolves both models under one frozen teacher: conditioned on motion, it distills the renderer into a block-causal student; unconditionally, it scores rendered rollouts against real videos, supervising motion by the video it produces. This lifts the fidelity ceiling from the motion generator onto the stronger renderer. The two models run as parallel causal streams, reaching 15.4 FPS at 1.3 s latency with no quality degradation.

Wed 9 SeptComputer Vision and Pattern Recognition
The gist
Making videos of talking heads in real time is tricky because methods that focus on quality usually need very big models and are slow, while faster methods don’t look as good. The authors found that mixing audio with the entire video information wastes effort because the audio mostly affects motion, not appearance or background. Their solution is to separate motion from identity and appearance, using a smaller model to predict motion from audio and a separate model to create video frames from motion. With a special training method balancing these models, they can generate high-quality streaming videos quickly and efficiently.
Open → 2609.10317v1