Group variance policy optimization improves large language model training stability
GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation
Artificial IntelligenceMachine Learning
Summary
Training large language models for specific tasks can be unstable because of certain math tricks used to weigh different training examples. The authors introduce a new method called Group Variance Policy Optimization (GVPO) that solves this by changing how the training updates are calculated, making the process more stable and flexible. Their approach also works well for teaching smaller models to imitate larger ones while learning from fresh data. This method provides a solid foundation for improving how large language models are trained and fine-tuned in practice.
What this means in practice
- •For machine learning engineers: Improve training stability and efficiency when fine-tuning large language models on task-specific data without relying on importance sampling.
- •For ai product developers: Develop more reliable distillation workflows for deploying smaller, efficient language models that maintain performance by mimicking larger models in real time.
Authors
Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, Hui Xiong
Abstract
Post-training plays a pivotal role in enhancing the reasoning capabilities and task-specific expertise of large language models (LLMs). Despite recent advances in post-training methods, such as Group Relative Policy Optimization (GRPO), their practical deployment remains impeded by training instability arising from the reliance on importance sampling. We introduce Group Variance Policy Optimization (GVPO), a novel post-training method that integrates the analytical solution of KL-constrained reward maximization into its gradient weighting scheme. This formulation provides an intuitive interpretation: GVPO's gradient corresponds to the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly to the KL-constrained reward maximization objective, and (2) it enables flexible sampling distributions without requiring importance sampling. Beyond general post-training, we show that GVPO naturally extends to on-policy distillation (OPD). Furthermore, GVPO enables the optimization of a broad family of extended OPD objectives, providing a principled foundation for diverse objective design. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training and on-policy distillation.