TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present a method called TaoMate for generating long videos with synchronized audio and visuals in real time. Their approach uses a special memory system that keeps important visual references and compresses past information to avoid forgetting while making the process faster and more stable. This helps the generated videos maintain consistent appearance and stay well synchronized with the audio over time without slowing down the generation. They tested their method and found it works well for keeping the character's look steady and matching speech and facial movements.
causal modelsaudio-visual synchronizationpersistent memoryautoregressive inferencevisual anchorsdenoisingvideo generationspeech conditioningattention mechanismtemporal consistency
Authors
Qijun Gan, Chenwei Zhang, Meiguang Jin, Junfeng Ma, Qiu Shen
Abstract
Real-time long-form digital-human generation relies on causal models to extend audio-visual content while preserving subject appearance and audio-video synchronization across successive segments. A bounded cache retains local motion and phonetic context but discards older evidence, whereas attending to the complete generated history is computationally expensive and can propagate accumulated errors. We present \method, an anchor-guided persistent-memory framework for few-step joint audio-video generation. The framework preserves an immutable visual anchor, compresses completed video and audio blocks into fixed-capacity dynamic states, and retrieves those states through modality-specific residual attention without extending the active cache. A reference-aware modulation method additionally conditions video features on dynamic and anchor appearance statistics. Anchor-preserving causal-context distillation varies rollout horizon, prefix provenance, and cache-history reliability while keeping the immutable visual anchor unperturbed. By separating persistent memory from stage-local denoising dependencies, \method further admits stage-parallel execution across blocks, accelerating autoregressive inference without pipeline-specific retraining. We evaluate long-form video continuations with appearance, temporal, synchronization, facial, and speech diagnostics. Results show that \method preserves stable appearance across prompt-conditioned segments and strong audio-visual synchronization under autoregressive generation. Our project page is https://taoliveaigc.github.io/TaoMate.