Adding options lowers multiple-choice scores without improving accuracy
Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation
Computation and LanguageArtificial IntelligencePerformance
Summary
Adding extra answer choices to multiple-choice questions can make people or machines score lower even if their actual knowledge isn’t worse. The authors studied 12,000 Turkish questions and added options copied from other questions, then tested 25 models to see how scores changed. They found that many correct answers were lost when extra, borrowed options were added, especially when questions were tricky or ambiguous. However, these lower scores do not necessarily mean the test measures knowledge better. Instead, they show the challenges in creating fair multiple-choice evaluations when options increase.
What this means in practice
- •For language model developers: Assess how adding answer options affects model accuracy on multiple-choice benchmarks in Turkish to avoid misleading performance drops.
- •For test designers: Understand the risks of adding extra answer choices that can reduce score accuracy without improving fairness in multiple-choice exams.
Authors
M. Ali Bayram
Abstract
Adding answer options can lower multiple-choice scores without improving assessment validity. Turkish MMLU Pro examines this distinction using 12,000 Turkish-source questions across 58 sections. Each question retains its stem, five original options and source key, and receives five options copied from other questions in the same section. Sentence-embedding retrieval proposes candidates; a language model selects existing identifiers. Deterministic verification reconstructs all 60,000 additions. A 25-model calibration exposes scoring and generation-budget effects. Five evaluations produce source-key accuracies of 34.8%-81.4%. On 981 shared questions, one API-served model falls from 93.7% with five choices to 83.1% with ten; 102 of 115 lost correct responses select borrowed options. The decrease is 24.4 percentage points on heuristically flagged negative stems and 5.9 points elsewhere. A completed human-checked audit of 200 sampled questions, with undocumented reviewer tool use, yields 47 and 31 multiple-answer judgments across the two record sets, 25 of the latter unresolved. These records support concern about ambiguity, while their dependence and incomplete reviewer-method documentation limit validation. Because order and labels also change, the paired comparison measures augmentation as implemented. The contribution is a traceable construction and an analysis of its validity limits, not evidence that lower ten-choice scores measure knowledge better.