Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
2026-06-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created MedSP1000, a new way to test medical AI by simulating realistic doctor-patient interactions using trained actors called standardized patients. Unlike traditional tests that only check one-step answers, this benchmark evaluates how well AI models manage a full patient visit, including gathering info, planning, and adjusting care over time. When tested, even the best AI models completed less than two-thirds of expert criteria, showing current AI isn’t reliable enough for real clinical use. The study highlights that ongoing, interactive testing reveals problems traditional tests miss.
Large Language ModelsStandardized PatientsClinical BenchmarkingMedical AIInteractive EvaluationPatient SimulationClinical Decision MakingRubric AssessmentGPT-5.5Clinical Agent
Authors
Cheng Liang, Pengcheng Qiu, Ya Zhang, Yanfeng Wang, Chaoyi Wu, Weidi Xie
Abstract
Large language models (LLMs) are increasingly proposed as clinical agents, yet static, single-turn benchmarks cannot capture how a model dynamically delivers care across an encounter: gathering information, planning treatment, and adapting longitudinal management across successive patient states. Medical education has long addressed an analogous challenge through standardized patients (SPs): trained actors who consistently portray clinical cases, enabling realistic practice and objective, scripted assessment. Here we introduce MedSP1000, an SP-derived interactive benchmark for clinical-agent evaluation, including 1,638 SP cases with 24,602 trajectory-level peer-reviewed rubrics. MedSP1000 converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubric. In each simulation evaluation run, a clinical agent interacts in closed loop with a patient agent and an environment controller, and its behaviour is scored throughout the encounter against expert criteria specified in the original materials. Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios. The best-performing model, GPT-5.5, completes only 60.4% of expert-defined rubric items, whereas the strongest medically specialized model reaches 40.0%; increasing test-time compute produces no measurable gain. These results suggest that current LLMs, including agentic systems tuned for medicine, are not yet reliable enough to be safely integrated into actual clinical practice. More broadly, MedSP1000 shows how process-level, SP-style evaluation can reveal clinically relevant failure modes that single-turn benchmarks miss.