LLMs predict medical exam question difficulty from their own responses
LLMs as Judges and Examinees for Medical Item Difficulty Prediction
Computation and LanguageArtificial Intelligence
Summary
Understanding how hard medical exam questions are is tricky just from reading their text. The authors studied large language models (LLMs) by making them act like test graders and also pretend to be students taking the tests. These roles let the models produce different clues about how difficult each question is. Combining these clues gave better predictions than previous methods on two big medical exam datasets. Their approach can be used locally on secure or unreleased exam questions.
What this means in practice
- •For medical test developers: Estimate difficulty of new or secret exam questions using LLM generated judgments and simulated responses without needing real test-taker data.
- •For educational assessment vendors: Integrate open-weight LLMs to predict question difficulty remotely for high-stakes medical exams, supporting test item management under security constraints.$Commercial implications: Enables development of commercial difficulty prediction tools for secure exam items using deployable open-weight LLMs.
Authors
Xiaomeng Xiong, Marcus Walker, Yanlin Jiang
Abstract
Predicting medical item difficulty from item text alone remains challenging. Prior work has largely relied on linguistic features or text representations derived from general-purpose and domain-specific language models, but predictive performance has remained modest. This study takes a different approach by using difficulty-related signals that emerge when open-weight large language models (LLMs) interact with test items. Using USMLE and PANCE medical licensure examination items, we examined LLMs in two roles. As judges, the models provided absolute difficulty ratings and pairwise difficulty comparisons. As simulated examinees, they generated response-based signals including correctness, keyed-option probability, response entropy, and the probability margin between the top two options. We then combined the strongest judgment and response signals with baseline item features in downstream prediction models. Pairwise difficulty judgment was the most consistent individual LLM signal across both datasets, while simulated-examinee signals and absolute judgments provided complementary information in some feature combinations. The best configurations reached a Spearman correlation of $ρ$ = .447 on USMLE, exceeding the best system in the 2024 BEA shared task ($ρ$ = .25), and $ρ$ = .494 on PANCE. These results were comparable to direct absolute judgments from the proprietary frontier model Claude Opus 5 ($ρ$ = .443 and .488, respectively). Overall, judgment and response signals from open-weight LLMs captured difficulty-related information beyond text representations. Because these models can be deployed locally, the approach offers a practical option for predicting the difficulty of unreleased or high-security test items.