Bias correction improves large language model evaluation with fewer comparisons
Accounting for Bias Enables Sustainable LLM Evaluation
Artificial IntelligenceMachine Learning
Summary
Evaluating large language models usually involves having these models judge which outputs are better, but this method often has hidden biases that skew results. The authors show that these biases, like favoring certain answer positions or lengths, can't be fixed just by running more tests. They introduce a new way to model and correct these biases to get reliable rankings from fewer comparisons. This method is also more efficient and trustworthy, making it a better way to evaluate language models over time.
What this means in practice
- •For machine learning engineers: Produce reliable LLM performance rankings using fewer evaluations while accounting for reviewer biases.
- •For ai ethics teams: Improve transparency and fairness in automated LLM evaluation by explicitly modeling known judgment biases.
Authors
Harshita Katoch, David Antony Selby, Gerrit Großmann, Sebastian Vollmer
Abstract
LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.