DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models
2026-08-10 • Machine Learning
Machine LearningComputer Vision and Pattern Recognition
AI summaryⓘ
The authors study a way to improve image generation models that use flow-matching. Normally, adapting these models to different tasks after training can be tricky and unstable. They propose DreOPD, a new method that combines reinforcement learning ideas with a more stable learning approach called on-policy distillation. DreOPD uses a clever trick with a 'degraded reference' to better guide the model's learning, leading to better performance across various tasks compared to previous methods. Their tests show DreOPD works well with one or multiple teacher models.
flow-matching modelsimage generationpost-trainingreinforcement learningon-policy distillationtrajectory optimizationreward extrapolationvelocity regressionmulti-task learning
Authors
Mingfeng Lin, Chengfei Cai, Lin Xu, Yuxiang Wei, Liang Han
Abstract
Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specific optimization objectives. Reinforcement learning enables direct optimization of task-specific rewards beyond the original models, yet trajectory-level optimization may incur high-variance gradients and cross-task interference. On-policy distillation (OPD) offers dense and stable supervision on student rollouts, but conventional teacher matching remains imitation-based. We propose DreOPD, a Degraded-reference extrapolative OPD method for flow-matching models that bridges these two paradigms. Our DreOPD converts implicit reward extrapolation into closed-form velocity regression, enabling extrapolative post-training with the stability of OPD. It further uses a mildly degraded reference to strengthen the teacher-reference contrast, yielding a clearer extrapolation direction. Experiments on single- and multi-teacher settings show that DreOPD outperforms OPD and multi-task RL baselines in average performance, while surpassing specialized teachers on most metrics.