LLM judge instructions and setup can change individual AI verdicts
Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals
Computation and LanguageArtificial IntelligenceComputers and Society
Summary
Sometimes computers that judge answers may give different decisions even if their overall scores look the same. The authors tested a popular AI judge by giving it the same answers many times but changing instructions and setup slightly. They found that while total scores stayed pretty steady, lots of individual decisions about which answer was better changed. This means it's important to check agreement on each question, not just overall scores, when using these AI judges.
What this means in practice
- •For ai development teams: Ensure AI answer evaluations include repeatability checks on individual verdicts, not just overall scores, to improve reliability of preference labels.
- •For ai service operators: Use repeated setups and prompt-level agreement reporting when monitoring large language model judge consistency to maintain trustworthy AI assessments.
Authors
Louise Yorke
Abstract
LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions. We show that similar aggregate scores can conceal substantial changes in individual verdicts, both when the same setup is repeated and when the instructions change. Using 500 Arena-Hard v2.0 prompts with two models' answers held fixed, we run GPT-4.1 as the judge under all eight combinations of criteria, output format and task framing, and repeat every setup unchanged at temperature zero on a pinned model version. Rerunning the default setup moves the aggregate score by about one point but changes the win/tie/loss outcome on 21.2% of prompts. Instruction changes introduce further sensitivity: in one setup, which asks for the verdict without reasoning and also drops several other instructions, outcomes change on 28.2 percentage points more prompts than the two setups' unchanged reruns do on average, and the aggregate score falls by 4.09 points. In another, where the criteria prioritise accuracy, outcomes change on 6.7 percentage points more prompts by the same comparison, while the aggregate score barely moves. A stable aggregate score therefore does not establish that individual verdicts are repeatable. This distinction matters when those verdicts are used as preference labels or to assess individual responses. Evaluations that rely on individual verdicts should repeat each setup they use and report both aggregate scores and prompt-level agreement.