Generative reward models improve rationale quality and reduce bias
Judging the Justification: Decision Certificate Supervision for Generative Reward Models
Computation and Language
Summary
Decision-making AI can explain why it makes certain choices, but sometimes these explanations don’t actually support the decision. The authors propose a new approach called DeCert that judges how good these explanations are without needing perfect example explanations to compare against. DeCert helps AI systems prefer explanations that better justify their choices, leading to clearer and more trustworthy reasoning. Tests show DeCert makes these explanations more consistent and reduces errors caused by bias.
What this means in practice
- •For ai system developers: Improve AI decision systems by training them to produce clearer and more accurate explanations that align with their decisions.
- •For customer support automation teams: Enhance chatbot responses to provide well-justified answers that can be trusted by customers without needing costly example explanations.
Authors
Guiquan Sun, Bing Bai, Martin Renqiang Min, Dongjin Song
Abstract
Generative reward models (GenRMs) generate natural language rationales to justify preference judgments, yet a correct verdict does not guarantee a sound justification. Existing approaches often rely on reference explanations to supervise rationales, but obtaining such references can be costly and does not necessarily establish whether generated rationales support the resulting verdict. To address these limitations, we propose DeCert, a framework that evaluates the validity and verdict consistency of generated rationales without requiring reference explanations. Crucially, DeCert translates these assessments into preference supervision that accounts for both verdict correctness and justification quality. In particular, when multiple sampled judgments share the same correct verdict, differences in certificate quality enable DeCert to construct rationale preference pairs that distinguish stronger from weaker justifications. Extensive empirical studies across two model families and multiple in- and out-of-distribution benchmarks demonstrate that DeCert consistently outperforms strong generative judge baselines while improving rationale quality. Moreover, DeCert effectively mitigates positional bias, improving position-consistent accuracy by up to 14.8 points. These results highlight the importance of supervising not only which verdict is correct, but also how that verdict is justified.