Which medical questions benefit most from detailed answer explanations
Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA
Computation and LanguageArtificial Intelligence
Summary
Medical question-answering systems often label correct answers, but detailed explanations (rationales) are rare and costly to produce. The authors studied how to pick which questions should get these explanations when there's a limited budget. They propose a method that identifies questions where explanations help improve answer consistency, especially when formatting changes. Their results show modest accuracy improvements overall but better robustness to changes in how questions are presented.
What this means in practice
- •For hospital data teams: Focus explanation annotation resources on medical questions that increase model stability under formatting changes, improving deployment reliability.
- •For legal compliance teams: Select question rationales that enhance answer robustness to ensure trustworthy explanations in regulated medical AI applications.
Authors
Yuexin Wu, Dayou Yu, Vasile Rus
Abstract
Medical question-answering datasets often contain answer labels, whereas high-quality rationales remain scarce, noisy, or costly to validate. This changes the acquisition question: rather than asking which questions should be labeled, we ask which already-labeled questions should receive rationale supervision under a fixed token budget. We study an offline version of this problem in which candidate rationales are visible to the selector but withheld from downstream training unless selected. We propose root-mean-square Robustness-based Sample Prioritization (RMS-RSP), which perturbs hidden states only at rationale tokens and measures the resulting shift in the gold-versus-best-distractor margin. Across five medical QA datasets, MedGemma-4B-IT, three training seeds, ten budgeted non-RSP selectors, and an unbudgeted full-supervision reference, RMS-RSP provides a deliberately qualified result. Its locked-budget accuracy is 60.61% on average versus 60.08% for Random, with a statistically resolved gain only on AfriMed-QA (+1.44 points). Its full-budget accuracy area is not better than Random. However, after three answer-option reorderings, RMS-RSP improves robust accuracy and semantic consistency by 1.91 and 2.85 points on average, respectively, with the same direction on all five datasets. Training on every pool rationale raises macro accuracy to 63.74%, but consumes 29--254 times more rationale tokens and does not uniformly improve robustness. These findings do not establish universal accuracy gains; they instead suggest that rationale-local boundary sensitivity can identify supervision that improves invariance to semantically equivalent formatting changes.