Human video animation preserves identity from one photo
One Image Is Enough: Self-Generated Supervision for Personalizing Human Video Diffusion at Test Time
Computer Vision and Pattern RecognitionGraphicsMachine Learning
Summary
Animating a person in a video using just one photo often fails to keep their face and clothes looking right when they move into different poses. The authors found a way to create fake video frames that mimic different poses of the same person based only on one image. These made-up frames help train a model to keep the person’s appearance consistent across movements, without needing real videos with multiple poses. Their method improves the natural look of animations better than using limited real footage and works well on popular datasets.
What this means in practice
- •For video content creators: Create realistic human animations from a single image without needing multi-pose videos to train identity preservation.$Commercial implications: Enables personalized avatar animation tools that generate high-quality, consistent videos from one user photo.
- •For virtual fitting rooms teams: Improve digital try-on models by generating consistent clothing animations from a single customer image across multiple poses.
Authors
Sadman Rakib Pinon, Adamo Orsini, Yanis Falaki, Linlian Jiang, Jacob Kingi, Yang Wang, Xinxin Zuo
Abstract
Pose-driven human video diffusion animates a person from a single reference image and a target pose sequence, yet it struggles to preserve identity, rendering the face and garments unfaithfully as the pose departs from it. Personalization is the standard remedy, but its supervision is missing: one image fixes identity at a single pose, while a multi-pose subject video cannot be assumed in practice. Instead, we synthesize the missing supervision. A domain-tuned, pose-conditioned generator produces pseudo-ground-truth frames at selected key poses. An inference-time reference-only sampler steers these candidates toward the subject and keeps the most identity-consistent. A per-subject low-rank adapter then trains on static clips of them, configured from the reference alone, with no subject video, ground truth, or labels. Although lower in fidelity, these synthetic frames supervise more effectively than real ones. Real frames as captured carry motion that erodes the temporal prior, while a static clip conveys appearance alone. On TikTok and UBC Fashion, the adapted model improves the base on every metric, inverts the identity-versus-motion trade-off, and matches or surpasses adaptation on five real frames despite using none. Code and weights are available at https://github.com/SadmanPinon/one-image-is-enough.