Open stack enables real-time interactive avatar conversations
AVTR-1: Open Stack for Real-Time Interactive Avatars
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Real-time talking avatars need to sync their mouth and head movements perfectly with speech to feel interactive, not just move fast. The authors built AVTR-1, a software system that uses a compact model to generate smooth avatar motions based on both people’s voices during a conversation. They made the system stream smoothly while syncing with external speech tools and measured how well the avatars match speech and listener reactions. AVTR-1 performs better than similar systems on visual quality and runs fast enough for real-time use on common GPUs. They also created a new way to check if the speaking person’s voice actually influences the avatar’s movements.
What this means in practice
- •For virtual assistant developers: Integrate avatars that speak and move synchronously with an external voice agent for more engaging real-time conversations.$Commercial implications: Makes possible interactive avatar-based assistants with real-time motion synchronized to speech for consumer and enterprise products.
- •For game engine creators: Create NPC characters whose lip and head movements synchronize dynamically with player or AI-generated speech during gameplay.
Authors
Artem Kravtsov, Dmitrii Ziganshin, Vsevolod Poletaev, Gleb Balitskiy, Anastasia Tikhonova, Egor Burkov, Vadim Lebedev
Abstract
Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live system must synchronize the model's output with speech from an external voice agent, schedule video frames for playback, and handle interruptions. We introduce AVTR-1, an open stack for real-time interactive avatar conversations, built around a compact 153M-parameter autoregressive flow-matching motion generator conditioned on both participants' audio. We adapt its audio encoder for streaming through self-distillation. The stack turns the model's chunk-based generation into a continuous, synchronized audio-video stream driven by an external voice agent, and we analytically derive its contribution to the user-facing latencies and validate the resulting bounds with two commercial voice agents. Further experiments demonstrate that AVTR-1 leads the compared dyadic systems on all reported visual-quality metrics and most conventional listening-motion metrics while remaining competitive in lip synchronization. Its inference runtime operates in real time on data-center and consumer GPUs. However, conventional listening metrics do not establish whether the paired speaker's speech contributes to generated motion. We therefore introduce the Reference-Based Directed Granger Gain (R-DGG), which measures the additional predictive information carried by speaker speech after accounting for listener history and speaker motion. R-DGG finds statistically supported predictive dependence for recorded listeners and all evaluated dyadic systems, but not for talking-head generators without paired audio or mismatched speaker-listener pairs. We release the model weights, renderer, and serving backend under component-specific licenses.