Large language models reduce position bias with scale and reasoning
On the Position Bias of Large Language Models: A Comprehensive Study
Computation and Language
Summary
Large language models can make mistakes by favoring answer choices based on their position rather than content. The authors studied this problem across different model sizes, languages, and tasks. They found that bigger models and a technique called Chain-of-Thought prompting help reduce this bias. Models designed to think through problems step-by-step show almost no position bias, though they require more computing power.
What this means in practice
- •For ai system developers: Improve evaluation fairness in multiple-choice tasks by using larger models with Chain-of-Thought prompting to reduce position bias.
- •For multilingual product teams: Build more reliable language tools across various languages by mitigating position bias through scaling and advanced reasoning models.
Authors
Yining Li, Huayang Li, Lemao Liu, Taro Watanabe
Abstract
Recent studies have shown that position bias remains a critical vulnerability in large language models (LLMs), undermining both the integrity of multiple-choice evaluations and the reliability of retrieval-augmented generation. However, existing understanding of this phenomenon is largely derived from studies with limited model coverage and English-centric benchmarks, leaving the behavior of frontier models and diverse languages underexplored. In this work, we conduct a comprehensive investigation of position bias across model scales ranging from billion-level to trillion-level parameters, covering both multiple-choice and multi-document tasks in five languages. Our experiments reveal that while position bias remains noticeable in small-scale models and Non-English languages, increasing model scale and employing standard Chain-of-Thought prompting significantly mitigate this issue. Moreover, we demonstrate that emerging \emph{thinking models} with extended reasoning capabilities are nearly immune to position bias, suggesting a costly but general approach toward resolving this vulnerability.