Who judges AI answers changes decisions by up to 18 percent
Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels
Computation and LanguageArtificial IntelligenceComputers and SocietyMachine Learning
Summary
Different families of language models used as judges give noticeably different results when they compare answers from other models. The authors found a strong bias where judges tend to prefer answers from their own model family more often. This effect changes the final decisions in about 18.5% of cases, showing that who evaluates AI outputs matters. They developed a method to measure this bias while controlling for answer quality and confirmed the results with various checks.
What this means in practice
- •For ai system evaluators: Adjust evaluation panels for language model outputs to reduce bias caused by judges favoring answers from their own model families.
- •For ai fairness auditors: Use controlled estimators to detect and quantify judge bias in automated AI output evaluation processes.
Authors
David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey, Elvis Gyasi Owusu, Nii-Nai Derrick Sowah
Abstract
Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwise design with 9,312 judgments. A common per-family statistic is strongly confounded with candidate quality and correlates with Bradley-Terry ability at r = 0.95. We derive a corrected estimator that holds the candidate family fixed and compares judges. All four families then show a positive same-family lift (3.4-8.4 percentage points), with global FPS 0.067 (95% CI [0.053, 0.084], permutation p = 0.0002). The effect remains under panel-based quality controls, an independent human-consensus anchor, and a float16 judging replication. Judge-side likelihood is closely related to the effect: adding likelihood advantage reduces the controlled coefficient by 61%, which we treat as descriptive attenuation rather than causal mediation. Position is a separate failure mode. Across the panel, 55.4% of AB/BA pairs reverse, and reversal above 50% is incompatible with a simple independent content-noise model. Relative to a family-balanced reference, panel composition changes 18.5% of pairwise outcomes. A complete reproducibility archive has been prepared for public release.