Synergies in training steps improve large language model reasoning accuracy
Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training
Machine LearningArtificial Intelligence
Summary
Training large language models (LLMs) often involves several steps done one after another, like teaching with examples, learning from rewards, and copying from a better model. The authors found that how these steps are combined really matters. For instance, a small initial training helps later copying work better, but strong reward-based learning can make copying less effective. They also showed that making the teacher model better helps the student model learn more when copying. Their results suggest that each training step should be chosen not just for how much it improves the model itself, but also for how it sets up the next step to work well.
What this means in practice
- •For machine learning engineers: Configure LLM training pipelines to optimize each step for better overall model reasoning accuracy and efficiency.
- •For ai product developers: Improve AI assistant reliability in complex reasoning by adjusting training workflows based on model compatibility insights.
Authors
Emre Can Acikgoz, Yang Li, Zeyu Leo Liu, Srijan Bansal, Dilek Hakkani-Tür, Shafiq Joty, Semih Yavuz
Abstract
Modern LLM post-training composes supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy distillation (OPD) into multi-stage pipelines, yet these stages are typically designed and evaluated in isolation. We show that this composition is consequential: a stage that improves the current model can make the next stage less effective. Through controlled experiments with Qwen3 models on math and science reasoning, we first characterize OPD across nine student-teacher pairs spanning 2x to 53x parameter ratios and show that OPD effectiveness depends on student-teacher compatibility rather than teacher scale alone. The surrounding stages of OPD reshape this compatibility in three ways: (1) A brief SFT warm-up improves subsequent OPD, while an RLVR-strengthened student regresses under distillation from the same teacher. (2) Adapting the teacher with RLVR raises downstream OPD accuracy in proportion to the capability it adds. Following these two interventions, we find that combining teacher adaptation and student warm-up alone raise average OPD accuracy from 29.2\% to 43.8\% (50\% relative improvement) after the same number of distillation steps, with additional preparatory training. (3) At comparable accuracy, OPD leaves a stronger initialization for downstream RLVR than SFT, with a gap that widens as RL compute scales. Our results suggest that each post-training stage should be chosen not only for the capability it adds, but for the learning interface it creates for the next stage.