Judge ensemble method improves scoring for AI training rewards
Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
Machine LearningArtificial IntelligenceComputation and Language
Summary
Scoring AI-generated answers is tricky because different judges often give very different scores. This paper shows that just picking one judge can cause big differences in results. The authors introduce a way to combine multiple judges into a team that weighs their confidence and ignores big disagreements. This combined approach scores better than any single judge alone on medical and science tasks, making training AI models more reliable.
What this means in practice
- •For ai model trainers: Improve AI training by using multiple judges with confidence-based weighting and disagreement filtering to get more reliable reward signals.
- •For medical ai developers: Enhance evaluation of AI responses in medical domains by combining diverse judge feedback to better capture nuanced quality.
Authors
Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell, Yutong Bai
Abstract
Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.