Generative reward modeling improved by human critique guided training
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Artificial Intelligence
Summary
Sometimes, AI systems learn what is good or bad by receiving scores, but these scores don’t explain why. This paper shows a way to teach AI models to generate helpful critiques, like detailed feedback, instead of just scores. The method uses a small number of real human critiques to train a helper model that guides the AI to produce better feedback even when only simple score data is available. The new approach helps AI systems learn more precisely what is right or wrong and improves their overall evaluation skills.
What this means in practice
- •For llm developers: Train large language models to produce more informative feedback signals using human critique-guided reward models that enhance model alignment.
- •For ai deployment teams: Improve the reliability of AI evaluation by integrating fine-grained critique assessments into reward modeling, leading to safer and more accurate outputs.
Authors
Xuancheng Li, Beining Wang, Haitao Li, Heng Wang, Yujia Zhou, Qingyi Pan, Blaze Chen, Yiqun Liu, Min Zhang, Qingyao Ai
Abstract
Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside preference judgments, providing finer-grained evaluation signals. Their effectiveness depends heavily on critique reliability. However, existing GRM training typically uses final preference correctness as outcome supervision. Because the preference outcome space is highly constrained, unreliable critiques can still yield correct outcomes and thus be reinforced. Recent work leverages human critiques for process supervision, but such critiques are scarce and are often reduced to scalar rewards, leaving their fine-grained evaluative information underutilized. We argue that evaluative criteria learned from human critiques can be generalized to broader outcome-only preference data. To this end, we propose \textbf{EnGRICH}, a GRM training framework that pairs the GRM with a training-time MetaCritic learned from a small set of human critiques. MetaCritic constructs response-specific rubrics and uses them to evaluate the evidence coverage and correctness of generated critiques. The resulting signals provide both process rewards for fine-grained credit assignment and structured guidance for exploring better critiques. During GRM training, MetaCritic is further optimized to generalize human-grounded evaluative criteria to outcome-only data. At inference, the trained GRM operates independently. Experiments across seven reward-model benchmarks show that EnGRICH consistently improves over competitive baselines, while further analyses validate the effectiveness of its core mechanisms.