Large language models struggle to value evidence properly in medical diagnosis
Can LLMs Value the Right Evidence? Evidence-Value Misalignment in Dynamic Medical Diagnosis
Computation and Language
Summary
Sometimes AI models can guess the right medical diagnosis even when they don't have enough good evidence, which can be risky for patients. The authors created a test called MedEVM where models get new information step-by-step and must decide when they have enough to diagnose. They found that many models either guess too soon, fail to recognize when evidence is enough, or get misled by bad clues. To fix this, they made a system called EVD-Harness that helps models double-check their evidence before deciding, which greatly improved their diagnosis accuracy.
What this means in practice
- •For clinical ai developers: Improve AI diagnostic systems by adding evidence verification steps to reduce errors caused by premature or misguided diagnoses.
- •For healthcare ai safety teams: Use evidence-value alignment metrics to predict and prevent unreliable AI diagnostic outputs in clinical settings.
Authors
Kehua Feng, Yunsheng Lu, Yitong Qiao, Tiantian He, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu
Abstract
A correct diagnosis reached from insufficient or misleading evidence can pose a clinical hazard, yet outcome-based accuracy may reward such lucky guesses. We call this mismatch between diagnostic decisions and the value of available evidence Evidence-Value Misalignment (EVM). To disentangle evidential grounding independently from diagnostic accuracy, we introduce MedEVM, a dynamic benchmarking environment comprising 1,050 cases across 24 disease systems. Observations arrive turn by turn, requiring models to continuously calibrate its decision by deciding whether to wait for more evidence or submit a diagnosis. Across 9 LLMs, four interesting patterns are observed. (1) Miscalibrated evidence tracking. Making a diagnosis often fails to calibrate evidence sufficiency, even in more capable models, and even worsens in reasoning mode. (2) Misaligned diagnosis submission. Confidence in the correct diagnosis often fails to ensure timely submission despite sufficient evidence. (3) Evidence order matters. Reordering the same evidence changes diagnoses even when model confidence remains similar. (4) Misleading evidence remains influential. Added misleading evidence redirects diagnoses even after prior evidence becomes sufficient. We further verify that EVM predicts errors and that preventing premature submission improves accuracy. These findings motivate Evidence-Verified Diagnosis Harness (EVD-Harness). It decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control. Across five LLMs, EVD-Harness improves accuracy by 12.0--51.1 percentage points while mitigating EVM-related failures. Our results demonstrate that verifying evidential support before submission can make diagnostic decisions more reliable.