New method finds what makes questions harder for language models
What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language
Computation and LanguageArtificial Intelligence
Summary
Figuring out why some questions are harder than others is important for testing AI language models. The authors created a way to automatically find and check explanations for what causes this difficulty. They looked at many questions answered by different models to measure difficulty, then asked an AI to suggest reasons for the difference between easy and hard questions. Their approach not only explains difficulty but can also predict it and change it by editing questions, showing these reasons really matter.
What this means in practice
- •For ai model developers: Improve evaluation tests by using automatic difficulty explanations to select questions that better measure model ability.
- •For test designers: Edit or design test questions based on discovered difficulty factors to create assessments with controlled challenge levels.
Authors
Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan
Abstract
Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.