ViDS: Video Diffusion Shader using 3D Face Tracking
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present ViDS, a method for animating portraits in videos by using 3D face tracking to keep a person's identity and expressions accurate. They create a 3D face model from a single image and then animate it using movements from another video. Their approach uses detailed geometric information and a video diffusion model as a kind of neural shader to generate lifelike animations. Compared to earlier methods, their technique offers better control of facial expressions and head pose while maintaining the original person's look. They also introduce a way to smoothly generate longer video clips with less visual glitches.
Video Diffusion3D Morphable Model (3DMM)Face TrackingPortrait AnimationNeural ShaderExpression ControlPose ParametersAutoregressive SamplingIdentity PreservationNormal Maps
Authors
Wenbo Ji, Davide Davoli, Zhe Chen, Liam Schoneveld, Matthias Nießner, Jiapeng Tang
Abstract
We introduce ViDS, a Video Diffusion Shader that leverages 3D face tracking for expressive and identity-preserving portrait animation. We first reconstruct the identity-specific 3DMM mesh from the reference image, and then animate it using expression and pose parameters from a driving video. Leveraging dense geometric cues from 3DMM normal maps, we employ a video diffusion model as a neural shader to synthesize lifelike portrait animations while preserving the appearance and identity of the reference image. We find that more accurate 3DMM tracking enables finer-grained expression control. We also introduce an autoregressive diffusion sampling process that extends generation beyond the model's native window while reducing discontinuities between adjacent clips. Compared with prior diffusion-based approaches for portrait animation that rely on landmark-based conditioning or implicit motion latents, our method achieves more detailed and consistent expression and pose control while faithfully preserving identity and appearance. Detailed ablation studies validate the effectiveness of our design choices. Project page: https://fusheng-ji.github.io/ViDS/