Training language model judges using feedback improves subjective evaluation
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Computation and LanguageMachine Learning
Summary
Deciding the quality of some tasks can be tricky and depends on what rules or criteria are used. The authors show that teaching language models to judge using detailed language feedback is better than just looking at the final answer. They introduce a method that pays attention only to certain parts of the feedback, improving how well the model judges things it hasn’t seen before. This new way of training judges works better in tests on subjective tasks without losing accuracy on clear-cut ones.
What this means in practice
- •For ai content moderation teams: Use improved language model judges to evaluate subjective content relying on nuanced feedback rather than final verdict accuracy alone.
- •For customer service automation developers: Deploy judges that understand detailed user feedback better for more nuanced automated complaint or support case evaluations.
Authors
Ilgee Hong, Changlong Yu, Zhenghao Xu, Xin Liu, Yuwei Zhang, Qin Lu, Bing Yin, Tuo Zhao
Abstract
We study training LLM judges from natural language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation criteria the judge invokes and how it weighs them. The dominant approach, outcome-supervised RL (e.g., GRPO), credits every token in the rollout with a single scalar determined only by the accuracy of the final verdict, providing no separate credit at the criterion-choice tokens and ignoring the rich language feedback (e.g., preference rationales) that naturally accompanies preference labels. Self-Distillation (SD) is one natural way to use this language feedback: the same model, conditioned on this feedback, acts as a teacher providing dense, position-level supervision. However, not all positions carry equally useful signal. Using the per-position entropy shift between teacher and student, we identify two regimes: context sharpening, where the teacher concentrates probability on a particular feedback-aligned criterion expression, and context spreading, where the teacher distributes probability across multiple feedback-aligned alternatives. We interpret these patterns as follows: sharpening encourages memorization of a particular criterion expression, whereas spreading promotes semantic understanding by preserving these alternatives. Motivated by this asymmetry, we introduce position masking based on the entropy shift that retains the lower tail of the entropy-shift distribution. Experiments show that masking higher-entropy-shift positions improves out-of-distribution generalization over naive SD. The resulting self-distilled judges outperform judges trained with outcome-supervised RL by 2-9 percentage points on the evaluated subjective subcategories, while remaining competitive on objective ones.