Distilled Reinforcement Learning for LLM Post-training
Machine LearningArtificial Intelligence
Summary
The authors explain that improving large language models after they are trained usually happens through reinforcement learning (RL) or on-policy distillation (OPD), but both have limits. RL struggles because it only gets broad feedback, making it hard to learn new things precisely, while OPD blindly copies a teacher model, which either adds little new information or gives bad guidance. They propose Distilled Reinforcement Learning (Distilled RL), a new method that uses detailed teacher feedback inside the RL training to better guide learning and selectively pass on new knowledge. Their experiments show Distilled RL is better than RL and OPD for learning from both similar and different teacher models.
Authors
Chen Wang, Zhaochun Li, Jionghao Bai, Yining Zhang, Hexuan Deng, Ge Lan, Yue Wang
Abstract
Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.