CatPortrait enables controllable animated video portraits with 3D face cues
CatPortrait: Taming Video Generative Models for Portrait Animation via In-context Conditioning
Computer Vision and Pattern Recognition
Summary
Creating realistic animated videos of people’s faces is hard, especially when you want to control how the face moves or speaks. The authors developed CatPortrait, a method that uses 3D face information to guide existing video generators without changing their internal design. This means CatPortrait can make videos where a portrait’s facial expressions match desired movements or speech more easily and flexibly. It works for animations driven by either video or audio inputs and performs well across different tests.
What this means in practice
- •For video production teams: Create controllable portrait animations from existing videos without redesigning video generative models.$Commercial implications: Allows producing custom animated video portraits efficiently for entertainment and advertising markets.
- •For mobile app developers: Integrate face-driven or voice-driven animated portraits into apps using pretrained video synthesis backbones.
Authors
Ziqi Ni, Zhou Fang, Shijian Wang, Rui Li, Yi Zhou
Abstract
Recent video generative models have shown strong capability in synthesizing high-quality and temporally coherent videos. Yet, adapting them for controllable portrait animation remains nontrivial, as existing methods typically rely on architectural changes or extra conditioning modules, increasing complexity and reducing compatibility with pretrained models. We present CatPortrait, a simple and effective framework that enables controllable portrait animation via 3D face guidance as in-context conditioning. CatPortrait preserves the original video generation backbone and injects structured 3D facial guidance into the input space through spatial concatenation, allowing pretrained image-to-video models to animate portraits without sacrificing their pretrained priors. CatPortrait supports both video-driven and audio-driven portrait animation within a unified framework. Extensive experiments demonstrate that CatPortrait achieves competitive or superior results across both settings.