Evaluating reliability of large language models as judges in NLP tasks
All Verdicts are Not Equal: Rethinking LLM Judge Reliability
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) are often used to judge the quality of language tasks, treated like reliable truth. This paper shows that these models can be inconsistent and sometimes give different or wrong answers depending on tiny changes. The authors create a new way to measure if judgments are trustworthy by checking if they are consistent, fair, and accurate at once. They find that evaluating by scoring full rubrics instead of simple win-or-lose choices makes the judgments more reliable.
What this means in practice
- •For nlp engineers: Ensure evaluation results using LLM judges are reproducible and less biased by adopting rubric-based scoring and reliability metrics outlined in this work.
- •For ai product managers: Design more trustworthy AI evaluation pipelines by incorporating the trustworthy verdict rate metric to detect and mitigate unreliable LLM judgments.
Authors
Vineet Kumar, Darshita Rathore, Anindya Moitra
Abstract
LLM-as-a-Judge is the standard paradigm for NLP evaluation, yet its systemic reliability remains poorly understood despite being widely treated as a deterministic ground truth. We present a comprehensive reliability audit, stresstesting six frontier models across four benchmarks, five prompt formats, two presentation orders, three sampling temperatures, and ten repetitions per condition. Our empirical analysis reveals severe vulnerabilities: verdicts change across identical replications at temperature zero, position-order swaps flip the majority of verdicts on challenging tasks, and the most deterministic judge achieves perfect consistency by trivially repeating incorrect verdicts, agreeing with ground truth only 51% of the time. To formalize these multi-faceted failure modes, we introduce the trustworthy verdict rate (T ), a unified metric capturing the joint probability that an evaluation is reproducible, order-invariant, and accurate. UsingT , we derive a theoretical upper bound on accuracy imposed by position bias and show that reliability is item-specific rather than modellevel. Finally, we demonstrate that shifting from pairwise win-rate to holistic rubric scoring improves trustworthiness more than any single-format prompting intervention, offering an actionable framework for robust NLP evaluation.