Clinical language models struggle to gather and assess patient information
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy
Computation and LanguageArtificial Intelligence
Summary
Language models in healthcare often get tested only on how well they diagnose diseases given complete patient information. But in real life, doctors first need to ask patients the right questions and decide which tests to order. The authors created KlinikeBench, a set of tasks with virtual patients to measure not only diagnosis accuracy but also how well models gather important clinical information. They found that while some models get the diagnosis right when given full details, they often fail to conduct a proper interactive clinical assessment. This shows a big gap between knowing the right answer and doing a realistic medical evaluation.
What this means in practice
- •For healthcare ai developers: Test and improve language models that interact with virtual patients for more realistic clinical questioning and assessment.
- •For medical simulation trainers: Create training environments that evaluate clinicians or AI systems on interactive patient history gathering and diagnosis steps.
Authors
Xueting Fang, Zehui Li, Yang Yang, Camilla Giovino, Shubh K. Patel, Shailly Prajapati, Vallijah Subasri, Caihua Shan
Abstract
Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and determine which examinations are needed before reaching a diagnosis. Diagnostic accuracy alone therefore cannot establish whether an agent gathered essential information or conducted an appropriate clinical assessment. Furthermore, existing benchmarks lack professional clinicians' verification. To address this gap, we introduce KlinikeBench, a benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria. More than 35 clinicians contributed to case authoring and benchmark evaluation. In an empirical study, clinicians gave simulated dialogues higher mean quality ratings than reference conversations, which is adapted from real conversation. In each task, an LM has a fixed budget of turns to communicate with the patient, ask about relevant history, request examinations, follow action constraints, and record a final diagnosis. We score these steps separately as well as together. Across 31 models and seven model families, the best-performing models (e.g., GPT-6-astra and Claude Opus 5) succeed on less than 30% of tasks, even though their diagnosis accuracy reaches 90.7%. Some models benefit from talking with the patient; others diagnose well from a complete chart but perform much worse in conversation. Overall, KlinikeBench provides a testbed for evaluating the full clinical encounter and reveals a substantial gap between diagnostic accuracy and performance in interactive clinical assessment.