DuoMatching improves video generation with frame and joint distribution matching
DuoMatching: Joint-Marginal Distribution Matching for Few-Step Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos step-by-step can cause problems like blurry frames or loss of meaning over time. The authors propose DuoMatching, a new method that helps video generators learn by also looking closely at each single frame through hints from image generators. This approach combines understanding of both the whole video sequence and individual frames to make videos look better and stay more meaningful. Their experiments show people prefer videos made this way over other methods most of the time.
What this means in practice
- •For video content creators: Create higher-quality, semantically meaningful short videos with improved frame clarity and motion preservation for media and entertainment.
- •For machine learning engineers: Enhance video generation models by incorporating joint and marginal distribution matching methods for better video and frame-level supervision.
Authors
Jiahao Zhan, Yan Wang, Yongrui Ma, Qunliang Xing, Ruchang Yao, Runtao Liu, Shijie Zhao, Tianfan Xue
Abstract
Streaming video generation has benefited from distribution matching distillation (DMD), which matches the joint distribution of video frames to a video teacher's approximation of the real video distribution. Although this joint matching mitigates drift during autoregressive rollouts, limitations remain in visual quality and semantic alignment. To address these limitations, we propose DuoMatching, a distribution matching framework that approximates the real video distribution through a unified joint-marginal formulation. On top of existing joint matching formulations, the additional marginal matching objective provides dedicated frame-level supervision from an image generator, transferring complementary visual and semantic priors from it. To apply this frame-level supervision in video generation, we introduce LatentBridge to resolve the latent representation mismatch between the video student and the image teacher. Latent Variation Sampling further distributes such frame-level supervision across distinct temporal segments, reducing redundancy. Experiments demonstrate that DuoMatching improves visual quality, composition, and semantic alignment while largely preserving motion dynamics. Human evaluations show overall preference rates above 80% against all evaluated baselines. The project page is available at https://johnzhan2023.github.io/DuoMatching/.