Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
2026-06-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors study a method called Group Relative Policy Optimization (GRPO) used to align flow-based generative models with human preferences. They identify a problem: these models normally require many steps to generate samples, which is costly, so existing methods use very few steps and get limited feedback. To fix this, the authors propose Pave-GRPO, which cleverly breaks down a few coarse steps into many finer sub-steps without extra cost, spreading reward feedback more widely in time. This helps the model learn preferences better and more precisely. Their experiments show that Pave-GRPO improves preference alignment effectively without extra sampling expense.
Flow-based generative modelsGroup Relative Policy Optimization (GRPO)Preference alignmentDenoising stepsPolicy-gradient updatesTemporal supervisionVelocity decompositionReward feedbackSampling budgetGenerative modeling
Authors
Pengyang Ling, Jiazi Bu, Yujie Zhou, Yibin Wang, Zhenyu Hu, Zihan Zhang, Yi Jin, Huaian Chen, Yuhang Zang
Abstract
Post-training via Group Relative Policy Optimization (GRPO) has emerged as a powerful paradigm for aligning flow-based generative models with human preferences. However, the iterative denoising nature of flow models incurs substantial costs when generating group rollouts for policy-gradient updates, compelling existing methods to train with extremely few denoising steps. This temporal sparsity severely restricts preference optimization: reward feedback can only reach a handful of stages per trajectory, leaving the vast majority of intermediate denoising steps without direct supervision and thus compromising alignment granularity. To address this, we propose Pave-GRPO, which reformulates the GRPO objective through Principled average velocity decomposition. Rather than generating expensive high-step rollouts, we maintain efficient few-step group sampling but decompose each coarse transition into an equivalent ensemble of finer sub-trajectories spanning multiple intermediate timesteps. This propagates reward feedback to a denser set of temporal stages for more comprehensive preference alignment without additional generation cost. This design offers two benefits: (i) zero-cost horizon expansion: through the direct reuse of piece-wise group samples and their associated rewards, Pave-GRPO significantly broadens the effective optimization scope under fixed sampling budgets; and (ii) comprehensive temporal supervision: by equivalently decomposing an instantaneous velocity target into a multi-timestep ensemble, it distributes reward signals across more intermediate stages of the denoising process, enabling finer-grained and more thorough preference optimization. Extensive experiments validate that Pave-GRPO effectively advances preference alignment across different reward settings, offering comprehensive performance enhancement.