Llm judges adapt to human preferences despite position bias
DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
Artificial Intelligence
Summary
Large language models can judge other AI responses but their opinions can be influenced by the order in which results are shown and differ from human choices. The authors introduce a method called DIAL that uses AI comparisons along with a small amount of human feedback to remove these biases and adjust the AI judgments to better match human preferences. They test this approach with many AI judges and human-labeled benchmarks and find DIAL creates rankings closer to what people want, even when the AI information isn’t perfect.
What this means in practice
- •For product review platforms: Improve automated ranking of reviews by reducing position bias and matching user preferences with limited human input.
- •For automated content moderation teams: Use calibrated LLM judges to prioritize content moderation decisions based on human-aligned preferences with fewer human labels required.
Authors
Zesheng Cai, Yingqi Fan, Sichang Chen, Jin-Hong Du
Abstract
Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human preferences.We introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.