Papers for

product review platforms

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Llm judges adapt to human preferences despite position bias

DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

Abstract: Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human preferences.We introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.

Fri 25 SeptArtificial Intelligence
The gist
Large language models can judge other AI responses but their opinions can be influenced by the order in which results are shown and differ from human choices. The authors introduce a method called DIAL that uses AI comparisons along with a small amount of human feedback to remove these biases and adjust the AI judgments to better match human preferences. They test this approach with many AI judges and human-labeled benchmarks and find DIAL creates rankings closer to what people want, even when the AI information isn’t perfect.
Open → 2609.31215v1