Real-time autoregressive video generation balances quality and speed
TriTD: Tri-Partite Trajectory-Distribution Distillation for Real-Time Autoregressive Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos using computers often requires balancing how good the video looks with how fast it can be made. The authors identify three main sources of errors that slow down this process or reduce quality. They propose a new method that combines three techniques to reduce errors and speed up video creation without losing quality. Their approach makes it possible to generate videos in real time on powerful hardware.
What this means in practice
- •For interactive media developers: Generate high-quality streaming videos in real time for interactive applications using a faster autoregressive generation method.
- •For computer graphics engineers: Improve training stability and output quality while speeding up video synthesis pipelines through trajectory-distribution distillation techniques.
Authors
Jia Li, Xurui Peng, Haowei Zhu, Tianyu Zhao, Jiexi Wang, Youwei Zheng, Yuxi Ren, Xinglong Wu, Xing Wang, Hayden Kwok-Hay So
Abstract
Autoregressive (AR) video diffusion has emerged as a leading paradigm for efficient video generation, yet its practical deployment remains bottlenecked by the trade-off between generation quality and inference efficiency. We attribute this bottleneck to three error sources in AR rollouts: the base model error, the train-inference distribution mismatch, and the upstream error propagation. Building on this decomposition, we derive closed-form upper bounds on the per-step and cumulative errors, providing a principled basis for jointly optimizing generation quality and inference efficiency. Guided by this analysis, we propose TriTD, a Tri-Partite Trajectory-Distribution co-optimized distillation framework with three complementary modules. First, for ODE initialization, we adopt a Next-Node Trajectory Distillation (NTD) objective, which shortens the supervision horizon and admits a lower regression-error floor. Second, we introduce Chunk-Graded Noising (CGN), a monotonically non-decreasing per-chunk noising schedule that helps narrow the expected train-inference noise mismatch and removes redundant KV-cache updates, yielding substantial speedups at no quality cost. Third, we propose Orthogonal-Parallel DMD (OP-DMD), a distribution-matching loss that retains the original DMD supervision while introducing an additional regression constraint in the orthogonal subspace to suppress off-axis drift, improving both training stability and final generation quality. Extensive experiments show that TriTD consistently surpasses existing baselines in visual quality while reaching a real-time throughput of 26.8 FPS on a single H100 GPU, delivering a balanced quality-efficiency solution for interactive streaming video generation.