Large language models reveal patterns behind medical exam question difficulty
What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty
Computation and LanguageArtificial Intelligence
Summary
Medical exam questions vary in difficulty, but it’s not clear why some are harder than others. The authors used large language models (LLMs) to study real exam questions from medical licensing tests and found certain patterns like misleading hints or the need to combine many clues. They showed that telling the models about these patterns helped them use difficulty ratings more effectively but didn’t always make their guesses more accurate. This approach can help people better understand what makes some medical questions tough.
What this means in practice
- •For medical test developers: Identify interpretable patterns that influence question difficulty to improve exam design and fairness.
- •For assessment platform engineers: Use rubric-guided difficulty labels to generate hypotheses for refining automated difficulty estimation tools.
Authors
Xiaomeng Xiong, Marcus Walker, Yanlin Jiang
Abstract
Research on medical item difficulty has largely focused on prediction rather than on identifying interpretable patterns associated with empirical difficulty. This study used empirical difficulty labels to guide large language models (LLMs) in inducing difficulty rubrics and examined how rubric guidance affected LLM difficulty judgments. Using training items and their empirical difficulty estimates from the United States Medical Licensing Examination (USMLE) and the Physician Assistant National Certifying Examination (PANCE), Claude Opus 5 induced separate rubrics for absolute and pairwise difficulty judgment. None of the induced rubrics treated surface features, such as stem length, option count, or images, as major sources of difficulty. Several patterns emerged across both examinations, including cue convergence, misleading cues, and option competition, although their specific forms differed. Retrieval of knowledge that could not be derived from the stem was particularly prominent in the PANCE rubrics, whereas the USMLE rubrics placed greater emphasis on integrating multiple clinical cues. We compared commercial and open-weight LLMs on held-out items under unguided and rubric-guided conditions. Rubric guidance changed how models used the difficulty scale but did not consistently improve performance. In absolute judgments, guidance encouraged broader use of the scale and generally improved coarse-grained agreement, without consistently improving rank ordering. In pairwise comparisons, its effects varied across models and examinations, and judgments became more sensitive to presentation order. These findings suggest that the value of label-guided rubric induction may lie more in generating interpretable, testable hypotheses from empirical difficulty data than in reliably improving difficulty judgments.