Reward models improve learning by combining specialist teachers effectively
RM-OPD: On-Policy Distillation for Generative Reward Models
Machine LearningComputation and Language
Summary
Training AI models to understand human preferences from mixed kinds of data is hard because different task types confuse the model and reward signals can be very sparse. The authors show a way to train separate specialized models for each task first, then combine their knowledge into a single model. They do this by teaching the combined model directly using detailed feedback from the specialists and controlling how much each specialist influences the combined model. This method beats previous approaches and sometimes the combined model gets even better than its teachers.
What this means in practice
- •For ai model trainers: Build more capable AI reward models by combining task-specific experts into one model more effectively.
- •For natural language processing teams: Improve multi-domain evaluation models by reducing interference between domain capabilities during training.
Authors
Qinglin Su, Licheng Pan, Zhen Fang, Jiakang Wang, Qingnan Ren, Yunsheng Lu, Yan Gao, Zhixuan Chu
Abstract
Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.