ViRDM improves few-step video generation without large teacher models
ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos step-by-step usually needs complicated setups with big pre-trained models and extra helpers to guide training, which can be slow and require lots of computer power. The authors figured out a new way called ViRDM that trains just the video generator itself by comparing what it produces to a target pattern, without the need for these extra models. This approach makes training faster, uses less memory, and produces higher quality videos in fewer update steps. Their method was tested and showed better results on a standard video benchmark using much less computing time.
What this means in practice
- •For video content creators: Generate high-quality short videos quickly using fewer computational resources during model training.
- •For graphics software developers: Integrate faster and more memory-efficient video synthesis that avoids large teacher and critic models into animation pipelines.
Authors
Zichong Meng, Chongjian Ge, Chun-Hao P. Huang, Yang Zhou, Huaizu Jiang
Abstract
Few-step autoregressive (AR) video diffusion enables low-latency streaming generation, but existing post-training methods predominantly rely on Distribution Matching Distillation (DMD), requiring both a large pretrained teacher and an online critic to estimate distributional discrepancies through diffusion scores. In this work, we ask whether this resource-intensive teacher--critic stack can be eliminated by post-training only the generator against a precomputed target distribution. Drawing inspiration from representation distribution matching (RDM) for one-step image generation, we systematically study its transfer to few-step causal video generation and identify three key barriers: a memory-intractable gradient path, a distinct video optimization regime, and representation distributions that underconstrain temporal dynamics. We introduce ViRDM, a teacher- and critic-free video post-training recipe that addresses these barriers sequentially. By coupling RDM with stochastically truncated clean-exit supervision, a lightweight VAE decoder, and staged vector--Jacobian products, ViRDM makes representation distribution matching memory-feasible for multi-step causal video rollouts. We further establish effective generated-population and initialization regimes for video RDM, and introduce lightweight dynamics regularization to compensate for the underconstrained temporal dynamics. ViRDM turns three-network distillation into generator-only post-training, reducing GPU memory use and training time while improving video quality. With only 20 generator updates, the recipe reaches 84.87 on the official VBench evaluation, outperforming the previous best few-step causal baseline by 0.36, while requiring 16 A100 GPU-hours. We additionally report exploratory results demonstrating the potential of the same recipe for lower causal sampling budget and for one-, two-, and four-step bidirectional generation.