TaoMate-HD speeds up streaming high-resolution digital humans and speech
TaoMate-HD: Progressive-Resolution Autoregressive Audio-Video Generation
Computer Vision and Pattern Recognition
Summary
Creating high-quality videos of digital humans who speak continuously is usually slow and can cause mistakes when new text is added. The authors designed TaoMate-HD, a model that builds videos in steps: first making a rough video at low resolution, then finishing with high detail, while producing matching audio throughout. This approach remembers what has already been said, so it avoids repeating or cutting off speech when the text updates. Their method runs faster and makes fewer errors in spoken words compared to earlier systems.
What this means in practice
- •For video game developers: Generate lifelike high-resolution digital characters that speak continuously and update dialogue smoothly as game events change.
- •For virtual assistant developers: Create digital avatars that can stream high-quality video and natural speech in real time, adapting seamlessly when new user inputs arrive.$Commercial implications: Enables selling advanced interactive digital assistants with realistic streaming visual and audio responses to consumers or enterprises.
Authors
Qijun Gan, Chenwei Zhang, Xinyang Li, Meiguang Jin, Junfeng Ma, Shen Qiu
Abstract
Streaming digital humans produce detailed video while continuing speech as new text arrives. High-resolution denoising is costly, and text updates can cause repeated or interrupted speech. We present TaoMate-HD, an autoregressive model for high-resolution audio-video streaming. For each short segment, or chunk, two denoising steps establish coarse video structure at low resolution. A final step refines the video to high resolution, with audio generated throughout the same trajectory. Each stage passes its audiovisual predictions to the same stage of the next chunk as history. Across text updates, the model retains this history to infer what has already been spoken. Continuation training utilizes asymmetric text conditioning, with separate text windows for generation and distillation. Experiments show a 1.6x speedup over direct high-resolution generation with the same model and hardware. Optimized stage-parallel deployment provides a further 1.9x overall speedup over serial progressive generation. Our continuation training reduces character error rate by 11.18 percentage points relative to the baseline. The model also supports hour-long audiovisual generation. Our project page is https://taoliveaigc.github.io/TaoMate-HD.