Video generation improved with faster few-step distilled models
VDOT++: Unified Few-Step Video Generation via Unbalanced Optimal Transport Distillation
Computer Vision and Pattern Recognition
Summary
Generating videos from text or images is usually slow because it requires many steps using big models. The authors improved a previous method by letting the smaller, faster model focus only on the important parts of the video, even when many different outputs are possible. They also made the model better at keeping details by changing how matches between frames are measured. This makes video generation quicker and still high quality across different ways of creating videos.
What this means in practice
- •For video content creators: Speed up text- or image-driven video generation while maintaining output quality with fewer processing steps
- •For mobile app developers: Integrate efficient few-step video generation models to enable video content creation on devices with limited compute power
Authors
Yutong Wang, Xingtong Ge, Enhuai Liu, Yunke Wang, Tianfan Xue, Yu Qiao, Yaohui Wang, Xinyuan Chen, Chang Xu
Abstract
Video creation spans text-to-video (T2V), image-to-video (I2V), and condition-based generation, yet video diffusion models remain costly because they repeatedly evaluate large backbones during sampling. Distribution matching distillation (DMD) reduces this cost, but its reverse Kullback--Leibler (KL) objective can provide unstable or incomplete guidance when the student and teacher distributions have limited overlap. VDOT addressed this issue by adding optimal transport distillation (OTD), whose explicit coupling supplies geometric directions for condition-based generation. Balanced OTD, however, performs full-mass matching between the spatial tokens of each corresponding student--teacher frame pair. This assumption weakens for T2V and I2V, where one condition admits many valid outputs and spatial content need not align across different realizations. We present VDOT++, a unified distillation framework that applies the same training recipe separately to generators for the three task families. It makes OTD robust to output diversity through an asymmetric unbalanced formulation that allows unreliable student tokens to carry less mass while maintaining coverage of the teacher tokens. An $\ell_1$ ground cost further replaces mean-based aggregation with a more mode-preserving weighted median that limits the influence of distant transport targets. The two changes respectively determine whom to match and how the selected targets should be aggregated. We additionally combine distribution matching and adversarial refinement through sequential backward passes, and exploit the decoupled score networks for cross-scale distillation, where larger score networks improve a compact generator. Experiments on UVCBench, VBench, VBench-I2V, and the VACE benchmark show that the resulting four-step generators are competitive with many-step teachers and strong few-step baselines across all three task families.