Pixel conditioned video diffusion enables streaming head avatar animation
PixReenact: Pixel-Conditioned Causal Video Diffusion for Streaming Head-Avatar Reenactment
Computer Vision and Pattern Recognition
Summary
Animating a still image of a person's head in real time based on live video input is hard because it requires keeping the person's identity stable while accurately copying movements, and it must happen quickly. The authors developed PixReenact, a new way to do this using a video diffusion model that looks at raw pixels instead of abstract motion or identity codes. Their method works continuously with low delay, handles difficult cases like unusual angles and facial expressions, and keeps the person's look consistent over long periods. This can help animate avatars smoothly as a person moves live.
What this means in practice
- •For virtual avatar developers: Create real-time animated head avatars that closely mimic live user expressions and motions with low latency and identity stability.$Commercial implications: Enables commercial avatar animation products for gaming and virtual meetings with smoother, consistent streaming animation.
- •For video conferencing engineers: Integrate continuous head avatar reenactment into live video streams to improve user privacy and engagement by hiding actual video with animated proxies.
Authors
Gavriel Habib, Dvir Samuel, Or Shimshi, Rami Ben-Ari
Abstract
Streaming head-avatar reenactment aims to animate a reference image according to a live driving video, requiring robust motion transfer, long-term identity stability, and low latency. Existing methods often rely on specialized identity or motion representations, which can discard useful visual information and inherit failure modes from external extractors. In addition, many recent diffusion-based reenactment methods use offline, clip-based generation, jointly processing and denoising an entire video clip before producing its output, making continuous low-latency streaming difficult. We introduce PixReenact, a pixel-conditioned streaming reenactment framework built on causal video diffusion. PixReenact conditions directly on VAE-encoded reference and driving frames, without specialized identity or motion representations. To separate reference identity from driver motion, we train with cross-identity pseudo supervision together with corrective objectives anchored to the original reference and driving inputs. Long self-rollouts reduce autoregressive drift, while state-aware dual-teacher distillation separately addresses cold-start and steady-state generation. Across three cross-identity benchmarks and a long-horizon streaming benchmark, PixReenact demonstrates robust cross-identity reenactment, particularly under challenging conditions such as extreme viewpoints, occlusions, and pronounced facial expressions, while maintaining the reference identity over long streams. A 4-NFE rolling student continuously emits four frames per update with a mean emission latency of 239 ms.