LLM judge consensus can exaggerate confidence in results
Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
Artificial Intelligence
Summary
When multiple AI models called judges agree on an answer, people often think the answer must be correct. But this paper shows that these AI judges often make similar mistakes because they are trained in similar ways. This means that their agreement isn't as strong proof of correctness as people think. The authors suggest using a small set of trusted examples to find shared errors and picking the best voting method before trusting these judges’ decisions.
What this means in practice
- •For ai evaluation teams: Improve system comparisons by accounting for shared errors among LLM judges to avoid overstating significance in results.
- •For machine learning engineers: Design evaluation protocols that use trusted example sets to calibrate judge accuracy and select appropriate voting methods.
Authors
Elias Hossain, Niloofar Yousefi, Ser-Nam Lim
Abstract
Consensus among LLM judges is often taken as strong evidence that a decision is correct. This assumes that judges make their errors independently. In practice, LLM judges are often trained and evaluated in similar ways, so they can make the same mistakes. We study how this dependency affects the reliability of consensus. We find substantial error correlation across both open-weight and frontier LLM judges. In our main bank of ten judges, the average pairwise correlation between judge errors is 0.21. As a result, the ten judges only provide roughly as much statistical information as 3.5 independent judges. The dependency is even stronger among the high-accuracy frontier judges we evaluate, including judges from different providers. In up to 28% of our comparisons, ignoring shared errors leads to the conclusion that one system is significantly better, while accounting for them does not. We also find that the pattern of errors matters. Errors shared by most judges and errors concentrated among a smaller group affect consensus differently and favor different voting methods. Measuring the overall amount of correlation alone is therefore insufficient. Our results suggest a simple approach: use a small set of trusted examples to estimate judge accuracy and identify shared mistakes. These shared errors should then be considered when analyzing the results, and the voting method should be chosen using trusted examples before it is applied to new data.