Large language models struggle to use long patient records for clinical decisions
ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making
Computation and LanguageArtificial Intelligence
Summary
Large language models have shown promise in answering medical questions when given the right facts. However, this paper shows that when these models need to make decisions based on a patient’s full medical history over time, especially for pregnancy care, they perform much worse. The authors created a new test with real pregnancy records to measure this and found that models often fail to find and use the right evidence from long health records. This suggests more work is needed to build reliable medical assistants for ongoing patient care.
What this means in practice
- •For hospital data teams: Improve evaluation metrics for clinical AI systems by testing their ability to extract evidence from complex, longitudinal patient records specific to maternity care.
- •For healthcare software developers: Develop more effective clinical decision support tools that better handle long patient histories by addressing evidence retrieval challenges identified in pregnancy EHR data.
Authors
Jun Xiang, Zhijie Bao, Rong Hu, Kaizhou Qin, Wei Chen, Zhongyu Wei
Abstract
The application of large language models (LLMs) to personalized medical assistants has garnered growing interest. However, existing medical benchmarks largely rely on static question answering with pre-selected evidence, leaving unclear whether LLMs can make reliable clinical decisions from real longitudinal electronic health records (EHRs). To bridge this gap, we introduce ObGynLongBench, a rule-grounded long-context EHR benchmark for obstetric and gynecologic decision-making, comprising 1,500 clinical decision-point cases from 976 real pregnancy EHR histories and traceable rules. Each case is anchored to a patient, a pregnancy-timeline point, and a pre-decision information boundary, enabling Evidence-only, Visit-level EHR, and History-level EHR evaluation. Evaluating 17 LLMs reveals a substantial Evidence-to-EHR Gap: models perform well when evidence is directly provided, but accuracy drops when evidence must be extracted from same-day records or full pre-decision EHR histories. Further analyses identify evidence utilization as a key bottleneck: performance decreases with longer EHR contexts and more complex evidence requirements, and earlier failures often predict later failures within the same patient history. Finally, active-search agents perform best among EHR access strategies, highlighting patient-specific evidence utilization as a central challenge for reliable personalized medical assistants. Resources are available at https://github.com/xiangjun2003/ObgynLongbench.