Video diffusion models create realistic 3d avatars from single images
Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars
Computer Vision and Pattern Recognition
Summary
Creating realistic 3D avatars from just one picture is tough because existing methods need detailed 3D data or models that can't capture all facial expressions well. The authors developed a way using video diffusion models, which have learned lots about how people look and move from video data, to make avatars that look very real and can be controlled in terms of viewpoint, face expressions, and body pose. They also fixed a problem where the expression controls were biased by camera angles, making the avatars consistent when seen from different views. Their method trains efficiently and achieves better results than previous state-of-the-art approaches.
What this means in practice
- •For game developers: Generate controllable and realistic 3D character avatars from a single photo for immersive game experiences.$Commercial implications: Enables creation of custom characters with photo-realistic detail and fluid motion for video games without extensive 3D scanning.
- •For virtual meeting platform teams: Produce photorealistic, multi-view consistent avatars for remote communication using only a single user image.$Commercial implications: Supports enhanced avatar realism and expressiveness that improves user presence in virtual meetings.
Authors
Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen, Matthias Nießner
Abstract
Video diffusion models (VDMs) trained on internet-scale data encode a rich prior on appearance, motion, and physics that is difficult to replicate with explicit 3D rendering and costly 3D data capture. We introduce Video Diffusion Avatars (VDAs), a method that creates photo-realistic, multi-view consistent avatars from a single image by fine-tuning a VDM for precise control over camera viewpoint, facial expression, and upper-body pose. Departing from 3D morphable models (3DMMs), VDAs use a learned expression space that lifts the expressiveness ceiling of 3DMMs, e.g., for the tongue and mouth interior. However, learned expression codes exhibit heavy viewpoint bias, as our DISTRACT benchmark reveals. We show that this bias can be removed through disentanglement, enabling their use as driving signals for multi-view consistent avatars. Camera and upper-body pose are controlled jointly via proxy-mesh renderings, giving independent control over expression, pose, and viewpoint. A camera-triplet training strategy makes the approach efficient: our models train for 34 hours on a single A100 using only public multi-view data. VDAs outperform state-of-the-art single-image 3D avatars on novel-view reenactment, reducing face LPIPS by 29% and improving perceptual quality (JOD) by 66% on BecomingLit. A static 3DGS reconstruction probe further confirms that our generations are the most multi-view consistent. Built on a video prior, VDAs produce effects beyond the reach of explicit renderers, including physically plausible hair dynamics, shadows, reflections, and refractions.