Improving autoregressive video generation with direct score correction and credit assignment
Learning the Autoregressive: Direct Score-Difference Estimation with Amortized Future Credit for Video Distillation
Computer Vision and Pattern Recognition
Summary
Training video generation models that create frames one after another is tricky because current methods refine the model indirectly and don’t properly consider how early frames affect later ones. The authors present Leta, a new technique that directly estimates the corrections from a teacher model and also assigns credit to earlier video chunks for their impact on future frames. This helps the student model learn faster and better from the teacher. Experiments show that Leta improves video generation quality compared to previous distillation methods.
What this means in practice
- •For video generation engineers: Enhance training efficiency and quality of autoregressive video generators by incorporating direct score-difference corrections and credit assignment.
- •For machine learning platform teams: Integrate joint teacher-guided update signals to streamline video generation model distillation workflows at scale.
Authors
Zixuan Duan, Yabo Chen, Xunzhi Xiang, Xin Zhang, Haibin Huang, Yin Chen, Chi Zhang, Qi Fan
Abstract
Few-step distillation for autoregressive (AR) video generation commonly first initializes an AR student and then refines it with distribution matching distillation (DMD). This refinement process faces two challenges. First, although initialization already yields a reasonably capable student, DMD obtains the refinement direction indirectly by estimating the full student score before forming the teacher-student correction. Second, detached-history training prevents the generator from assigning credit to a current chunk for its influence on subsequent chunks. We introduce Leta, a few-step distillation framework that addresses these challenges with ScoreDelta and FutureCredit. ScoreDelta directly estimates the teacher-student correction from denoising residuals, while FutureCredit learns an auxiliary cross-chunk update signal from vector-Jacobian products (VJPs) of next-chunk teacher residuals. Leta combines these signals into a joint teacher-guided update field in a single forward pass during generator updates. Teacher evaluations and credit VJPs are used only for estimator training. For ScoreDelta, we establish population equivalence to score-difference estimation and a conditional approximation advantage under limited capacity in a fixed-feature linear model. In our controlled experiments, Leta improves over DMD by 0.9 points on VBench and 0.66 on VideoAlign. This advantage persists at a higher generator update frequency (a generator-to-estimator update ratio of G:E=1:1), where Leta improves over DMD by 1.6 on VBench and 0.35 on VideoAlign. Project page: https://scdzx.github.io/Leta/.