Constrained-option scoring can misread model answers in multiple-choice tests
Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
The paper shows that some scoring methods for multiple-choice questions wrongly evaluate model answers when the model starts to provide answers outside the given options. The authors demonstrate this issue using a prompt that asks a model to predict whether another model would answer correctly. They introduce ways to detect when the scoring is failing and show that adding a prefilled answer stem improves the evaluation. Their findings help explain why some scoring approaches perform no better than guessing in particular tests.
What this means in practice
- •For machine learning engineers: Improve evaluation methods for multiple-choice tasks by detecting when scoring ignores likely answers outside fixed option sets.
- •For ai system testers: Use prefilled answer stems to get more reliable success forecasts when testing language models with constrained option prompts.
Authors
Jiaxuan Guo, Kejia Zhang, Shuo Xin, Jingxin Yang, Youran Sun, Haizhao Yang
Abstract
Constrained-option scoring reads a model's probabilities for a fixed set of permitted answers, so it returns a score even when the model is about to write something else. We study a prompt that quotes a multiple-choice item, ending in the item's own answer instruction, and asks for a forecast of whether a reader model given only a short note will answer it correctly. The model can begin answering the quoted item: on three Qwen3 checkpoints an answer letter is the most probable token on nearly every passage, and the forecast ranks the reader's correctness indistinguishably from chance. Beyond prior work's argmax check and prefill repair, we contribute a label-free diagnostic of the declared options' support, their first-token mass in context; a deletion test tracing the Qwen3 answer-letter start to the quoted instruction; and rendering and tokenisation checks for dropped separators and coinciding first tokens. Prefilling an answer stem lifts the median option mass from below to above one half on all nine checkpoints scored with this prompt and, with options and renormalisation unchanged, raises the Qwen3-averaged AUC from 0.4929 to 0.6152. lm-polygraph's default P(True) estimator shows a related failure: it reads True where Qwen3 without thinking favours an answer letter (MMLU) or the prompt's (A)/(B) label (TriviaQA), and its Qwen3-averaged AUROC is indistinguishable from chance on MMLU and below chance on TriviaQA. Scoring and renormalising the (A)/(B) labels after an answer stem raises it to 0.632 and 0.868, renormalising True against False in place reaches 0.660 and 0.860, and on TriviaQA both exceed all fourteen default single-answer library estimators. The original forecast is renormalised too, yet stays indistinguishable from chance: option mass flags low support in both settings without labels, and only checking the intended target shows which score still ranks.