Llm accuracy drops as answer choices grow in number
Overwhelmed by Choice: Studying LLM Decision Making at Scale
Artificial IntelligenceComputation and Language
Summary
Choosing the right answer from many options is harder for large language models than previously thought. The authors found that as the number of answer choices increases, the models' accuracy drops significantly. Two main problems cause this: the model becomes less confident in the correct answer and early preferences for answers are hard to change. They tested new ways to organize and evaluate answers that helped improve accuracy with lots of choices.
What this means in practice
- •For natural language processing teams: Improve multiple-choice question answering systems by incorporating hierarchical and permutation-based methods to handle larger candidate sets more accurately.
- •For clinical decision support developers: Enhance medical AI tools by addressing accuracy drops when many diagnosis or treatment candidates must be compared.
Authors
Yu-Chi Lin, Aryan Seth, Anshul Aravind, Eugene Lee, Tanmay Parekh, Nanyun Peng, Kai-Wei Chang
Abstract
Multiple-choice and candidate-selection evaluations are widely used to assess LLM reasoning and decision-making, yet most benchmarks contain relatively small candidate sets. It remains unclear whether conclusions drawn from these settings remain valid as the candidate space scales. We systematically evaluate LLMs as the number of competing candidates increases and find substantial accuracy degradation across tasks, prompting strategies, and model scales. Controlled analyses show that standard long-context retrieval explanations cannot fully account for this degradation. Instead, we identify two systematic failure patterns. First, gold-margin collapse: the score gap between the correct answer and the strongest distractor progressively shrinks, driven primarily by weakening confidence in the correct answer. Second, earlier candidate preferences become increasingly difficult to overturn, with later candidates exerting progressively weaker influence on the final prediction. Motivated by these findings, we evaluate hierarchical partitioning and permutation-based inference, which improve accuracy by roughly 20 percentage points at $N=160$ on both HotpotQA and MIMIC. Overall, our results identify candidate-set scale as an important evaluation-protocol variable and show that strong small-option performance does not necessarily imply robust large-scale candidate comparison.