Internal signals help large language model judges correct biased verdicts
Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
Artificial Intelligence
Summary
Large language model (LLM) judges sometimes pick the wrong answer because they rely too much on obvious features like length or position. The authors found that even when the judge picks wrong, the information to choose correctly is often inside the model's inner workings. By looking carefully inside the model, they can recover these hidden signals and get much better judgments. This approach lets the judge spot when it is likely wrong and improve the quality of labels for training preference-based systems.
What this means in practice
- •For machine learning engineers: Improve label quality and error detection when using LLM judges to evaluate and rank generated answers.
- •For ai system builders: Develop more reliable preference models by extracting hidden preference signals from LLM judge activations beyond their final verdicts.
Authors
Sourabrata Mukherjee, Sunayana Sitaram
Abstract
LLM judges are widely used to evaluate model outputs, but their verdicts can be unreliable: a judge may favor the worse answer for its position, length, or other surface features. When a judge is wrong, is the information needed to judge correctly absent from the model, or present in its internal representations but not reflected in the output? We study this across 64 open-weight evaluators and 14 datasets, including causal interventions on 41 judges (editing activations mid-run to see whether the verdict changes). On LLMBar, built so the superficially better answer is the worse one, the verdicts of 50 judges agree with human labels only 0.456 of the time, even after averaging both answer orders. Yet a small probe on the same judges' activations, with no weight updates, reaches 0.846, and 0.686 once surface features such as length and position are residualized out (0.507 with shuffled labels). The gap holds across eight benchmarks and model families, but is not universal: a score of how well surface features alone predict the human label, computed before any probe is trained, predicts the size of the gain (Spearman rho = 0.90). On rubric tasks that score one answer at a time, leaving no surface cue to exploit, reading the internals gives no advantage. The interventions also show that editing activations mid-network already changes the verdict, before it can be read off directly, and locate the pathways carrying position and length bias. At the same label budget, the recovered signal lets a judge flag cases where it is likely wrong and yields better labels for preference learning. A wrong verdict, then, does not mean the judge lacks the information, and a simple diagnostic shows when it is worth recovering.