Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
2026-06-02 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors introduce Skill Reward Model (Skill-RM), a new system that improves how reward models give feedback to large language models after training. Instead of using many different fixed rules or checklists, their method treats reward evaluation like a smart task that picks and combines the best evidence for each case. This makes the evaluation more flexible, consistent, and clear. Tests show that Skill-RM works better than older methods in different reward-related tasks and helps with reinforcement learning.
Reward ModelsLarge Language ModelsReinforced Fine-TuningReinforcement LearningReward EvaluationSkill Reward ModelDynamic Evidence AggregationBest-of-N SelectionAgentic TaskPost-Training Feedback
Authors
Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu, Qinliang Su, Xiaoxi Jiang, Guanjun Jiang
Abstract
Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines. However, current reward evaluation relies on heterogeneous criteria such as rule-based verifiers, ground-truth references, procedural checklists, and complex rubrics, where a unified mechanism to integrate all types of evidence remains unexplored. To this end, we propose Skill Reward Model (Skill-RM), a unified framework that reformulates reward modeling as the execution of a reusable Reward-Evaluation Skill. By treating reward computation as a structured agentic task, Skill-RM provides a consistent interface to orchestrate heterogeneous resources, dynamically selecting and aggregating evidence tailored to the specific requirements of each input. This approach enables the reward model to move beyond static evaluation, ensuring consistency and transparency across diverse tasks. Extensive experiments on reward benchmarks and downstream applications, including best-of-N selection and reinforcement learning, demonstrate that Skill-RM consistently outperforms traditional judge baselines. Our findings suggest that Skill-RM not only provides a unified solution for reward modeling but also achieves superior performance through the strategic and dynamic orchestration of evidence. The code is at https://github.com/Qwen-Applications/Skill-RM.