Large language models struggle to judge when medical data is sufficient
When to Stop Asking: Factor Coverage Bias and LLM Determinability Judgments with Underspecified Information in Medicine
Artificial Intelligence
Summary
Medical decisions often start with incomplete information, and it's important to know when you have enough details to decide safely. The authors found that large language models tend to rely too much on how many relevant factors they see, rather than whether the missing information would actually change the decision. This leads to errors like stopping too early or asking for too many unnecessary tests. Their study shows this bias is different from just reacting to longer text and provides a way to check if models make reliable judgments when information is limited.
What this means in practice
- •For clinical ai developers: Improve AI tools that assist doctors by detecting when medical information is truly sufficient for decisions to avoid premature conclusions or unnecessary tests.
- •For healthcare software teams: Use the evaluation benchmark to test and refine medical language models for better trustworthiness under incomplete information conditions.
Authors
Yusuke Watanabe, Yohei Kobashi, Taisei Tosaki, Takeshi Kojima, Yusuke Iwasawa, Yasushi Okuno, Yutaka Matsuo
Abstract
Clinical decision-making often begins with underspecified information and requires not only gathering missing information but also knowing when to stop. This calls for judging determinability (whether the available information is sufficient for a decision) by considering whether missing information could change the conclusion, not simply the amount of available decision-relevant information. Reliance on this amount as a cue for determinability, which we term \textit{factor coverage bias}, could undermine the trustworthiness of large language models (LLMs) through premature conclusions or excessive information gathering, such as unnecessary invasive tests. Yet whether LLMs exhibit this bias remains insufficiently studied. To address this gap, we construct a controlled pairwise benchmark in which LLMs identify the determinable case within each determinable--undeterminable pair. We add a decision-relevant factor to either side while keeping each case's determinability unchanged, and similarly add irrelevant text to distinguish this bias from context-length effects. Adding a relevant factor made LLMs more likely to select the augmented case as determinable even when it remained undeterminable, an effect that persisted with increased reasoning effort, while adding irrelevant text made them less likely to select the augmented case, an effect that reasoning substantially reduced. These findings identify factor coverage bias as a failure mode distinct from context-length effects and a potential risk to LLM trustworthiness. Our work provides an evaluation framework for LLM trustworthiness under underspecification.