AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model
2026-08-17 • Graphics
GraphicsComputer Vision and Pattern RecognitionMultimediaSound
AI summaryⓘ
The authors introduce AnyTalk, a new way to create 3D talking animations for any character without needing special animation data. They use a video diffusion model, which they fine-tune on images of the character with no motion, so the model keeps its ability to generate realistic movements. Then, they convert the generated talking videos into 3D animations by estimating the character's facial movement parameters. Their method works with different face types and blendshape setups and can run in real-time with a simplified version called AnyTalk_RT. This approach makes it easier to create lip-synced speech animations for many characters without much manual work.
3D speech animationvideo diffusion modelfine-tuningblendshape parameterstalking-head videolip-synccharacter rigginganimation datareal-time animationspeech-driven animation
Authors
Kwan Yun, Serin Yoon, Sunjin Jung, Jung Eun Yoo, Inyup Lee, Junyong Noh
Abstract
We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (\textit{CsF}) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, $\text{AnyTalk}_{RT}$, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.