Summary
Evaluating texts generated by AI language models often uses another AI as a judge, but it's unclear what information the judge needs to see to make good decisions. The authors tested different judges and what they were shown on various tasks, finding that the content shown to the judge mattered more than which judge was used. When judges don’t get to see important information like constraints, the feedback can cause AI systems to get worse with repeated rounds. They also found small parts of the data often do most of the work, and simple methods can sometimes match complex AI judges in spotting errors. The paper reports many unexpected findings and suggests guidelines for how to design AI judging systems.
What this means in practice
- •For ai development teams: Improve feedback loops by showing AI judges full constraints to avoid degrading model performance during iterative improvement.
- •For automated content reviewers: Use small focused tools like entailment graphs to detect errors in text efficiently alongside or instead of expensive AI judges.
Abstract
Evaluation of text generated by large language models (LLMs) is increasingly delegated to LLMs acting as judges, yet what such a judge must be shown remains unresolved. We treat a judge whose reports are acted on as a regulator, in the sense of cybernetics, and ask what it must be shown, how much, in what form, and what its reports must get right. On seven public benchmarks we varied only what the judge was shown. Across three judges whose prices differ 10- to 17-fold, what the judge was shown mattered more than which judge it was. When a system revised its answers on feedback from a judge denied the material they had to respect, it ended below its starting point on instruction following and question answering. On instruction following the loss grew with each round, and it persisted when the judge saw the user's request but not the constraints and with a second judge family. On news summarisation, a correct reference summary beside the source undid most of the source's contribution, whereas on single-answer questions a reference helped. About a tenth of the source did almost all of its work, and a small entailment graph matched an LLM judge at finding errors in grounded text. We committed 83 testable clauses before running; of the 81 that ran, 32 met their bar and 50 went in the predicted direction, and most quantitative predictions from the theory failed. We report every shortfall and derive design rules tied to measured effects.