RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction
2026-08-06 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors explain that new reward models that generate responses rather than just ranking them haven't worked well with reinforcement learning (RL) because RL expects simple numerical scores, not comparisons. They introduce a method called Ranking-based Reward Construction (RRC) that turns these comparisons into rewards RL can use. RRC uses two ideas: comparing responses against each other and comparing them to a small set of reference responses. Their tests show that RRC helps RL learn better from generative reward models.
reinforcement learningreward modelinggenerative modelscomparative rankingself-competitive rankinganchor-guided rankingresponse rankingreward constructionopen-ended chatreasoning benchmarks
Authors
Chenglong Wang, Ziming Zhu, Yifu Huo, Bei Li, Qiaozhi He, Yan Ding, Xiaoyang Hao, Yuxin Gao, Tianhua Zhou, Xiaojia Chang, Tongran Liu, Jingbo Zhu
Abstract
Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.