Kandinsky generates realistic talking avatar videos from speech
Kandinsky S2V 1.0: Audio-Driven Avatar Video Generation
Computer Vision and Pattern Recognition
Summary
Creating videos of realistic talking avatars is hard because the video must match speech closely in lip movement, expressions, and gestures. The authors introduce Kandinsky S2V 1.0, a system that makes long video clips of avatars from just one image, an audio recording, and a text prompt. Their models learn from over a million clips, producing videos where the avatar’s lips and gestures match the speech well. They also made a faster version that works much quicker but keeps the same quality. The team shared their code and models openly to help others build on their work.
What this means in practice
- •For video game developers: Create realistic character lip-sync and gestures from voice recordings to enhance game immersion without manual animation.
- •For virtual assistant developers: Generate lifelike talking avatar videos from speech input to improve user engagement in voice-based assistants and chatbots.$Commercial implications: Enables production of customizable, expressive avatar videos for virtual assistants, a product offering that improves user interaction and branding.
Authors
Nikolay Semenov, Artem Riabinin, Sergey Bassenko, Anna Averchenkova, Julia Agafonova, Anastasia Aliaskina, Kirill Kozlov, Lev Novitskiy, Nikita Samsonov, Nikolai Gerasimenko, Denis Dimitrov
Abstract
We present Kandinsky S2V 1.0, an open family of speech-to-video models for controllable audio-driven human animation, comprising Kandinsky S2V 1.0 Lite (2B parameters) and Kandinsky S2V 1.0 Pro (21B parameters). From a single reference image, a speech waveform, and a text prompt, the models generate long-form 25 FPS video in which lip motion, expression, pose, and gesture follow the speech. The architecture draws text and audio conditioning from one Qwen2.5-Omni model and adds a reference-aware three-scale visual memory. Training starts from the Kandinsky 5.0 Video foundation models [1] and combines mixed reference-memory layouts with an area-adaptive face loss. The training corpus contains 1.17M single-speaker clips selected by active-speaker detection, with captions generated directly in English and Russian. A side-by-side human evaluation against the strongest open-source baseline and two proprietary commercial models shows that Kandinsky S2V 1.0 Pro performs comparably to or better than these models on average and is also preferred for lip synchronization. On common speech to video benchmarks, Kandinsky S2V 1.0 Lite and Kandinsky S2V 1.0 Pro rank among the top performing models according to mean rank over five reported metrics. Beyond the base models, we provide π-Flow distilled variants of both Kandinsky S2V 1.0 Lite and Kandinsky S2V 1.0 Pro. The distilled Kandinsky S2V 1.0 Pro model generates video in 16 sampling steps without classifier-free guidance, achieving a 5.4 times end-to-end speedup over the base model, while matching or exceeding it on nearly all reported benchmark metrics. To accelerate open research and deployment in speech-to-video generation, we release the code, model checkpoints, and diffusers integration under the MIT license.