Large language models compared for virtual medical patient interactions

A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform

Human-Computer Interaction

Summary

Medical students practice talking to patients to learn how to diagnose and treat illnesses. The authors compared four different AI models that act like virtual patients to see which ones respond most like real human experts. They tested detailed conversation skills using many evaluations and found some models were closer to human behavior than others. They also built an online platform to let students practice with virtual patients safely and collect useful data.

What this means in practice

  • For medical training centers: Use the platform and virtual patients to provide scalable, realistic OSCE training and assessment for clinical communication skills.
  • For ai developers for healthcare: Improve virtual patient models by targeting specific weaknesses identified in different LLMs’ clinical dialogue performance.

Authors

Yongjin Yi, Jinyong Park, Sejoong Kim

Abstract

The Objective Structured Clinical Examination (OSCE) is widely used to assess clinical communication skills in medical education, requiring trainees to take histories and reason clinically with standardized patients (SPs). Large language model (LLM)-based virtual standardized patients (VSPs), typically built by persona-injected prompting, offer a practical alternative. However, existing evaluations rely on a single backbone LLM, small human rater panels, or LLM-as-a-judge scoring without an utterance-aligned human-expert reference, and so cannot identify the specific shortcomings of individual models. We evaluated four LLMs (GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B, and HyperCLOVA X), spanning commercial, open-source and Korean-developed models, against a human-expert reference across three Korean OSCE cases (hematuria, fever in pregnancy, and chronic cough). In a double-blind, fully crossed design, 17 senior medical-student raters scored every model utterance on three sentence-level and six encounter-level Likert metrics, yielding 7,480 data points. We analyzed the scores using a stratified linear mixed-effects model based on an 11-category history-taking and 3-level linguistic-form taxonomy. At the encounter level, three of the four LLMs fell 0.08 to 0.47 Likert points below the human-expert reference on Comprehensibility, Consistency and Hallucination-freeness. The difference was not statistically significant for Hallucination-freeness in Claude 3.5 Sonnet and GPT-4o, Consistency in GPT-4o, and Engagement in HyperCLOVA X. Llama-3.1 70B fell significantly below the reference on all six metrics. We also introduce a web-based OSCE platform that supports virtual practice across diverse clinical cases and continuously collects student-VSP dialogues under research and privacy consent.