Large language models struggle to use long medical records properly
ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making
Computation and LanguageArtificial Intelligence
Summary
Making good medical decisions from a patient’s full health record is hard for AI systems. The authors created a new test called ObGynLongBench to see how well large language models (LLMs) can handle real pregnancy health records over time. They found that these models do well when given clear pieces of evidence but have trouble when they need to dig through longer, more complicated health histories. The study found that the key challenge is how well models use and search for patient-specific information. This shows that AI still needs to improve to become a reliable medical assistant for ongoing care.
large language modelelectronic health recordlongitudinal dataclinical decision-makingobstetricsgynecologymedical benchmarkevidence utilizationpersonalized medicineactive search
Authors
Jun Xiang, Zhijie Bao, Rong Hu, Kaizhou Qin, Wei Chen, Zhongyu Wei
Abstract
The application of large language models (LLMs) to personalized medical assistants has garnered growing interest. However, existing medical benchmarks largely rely on static question answering with pre-selected evidence, leaving unclear whether LLMs can make reliable clinical decisions from real longitudinal electronic health records (EHRs). To bridge this gap, we introduce ObGynLongBench, a rule-grounded long-context EHR benchmark for obstetric and gynecologic decision-making, comprising 1,500 clinical decision-point cases from 976 real pregnancy EHR histories and traceable rules. Each case is anchored to a patient, a pregnancy-timeline point, and a pre-decision information boundary, enabling Evidence-only, Visit-level EHR, and History-level EHR evaluation. Evaluating 17 LLMs reveals a substantial Evidence-to-EHR Gap: models perform well when evidence is directly provided, but accuracy drops when evidence must be extracted from same-day records or full pre-decision EHR histories. Further analyses identify evidence utilization as a key bottleneck: performance decreases with longer EHR contexts and more complex evidence requirements, and earlier failures often predict later failures within the same patient history. Finally, active-search agents perform best among EHR access strategies, highlighting patient-specific evidence utilization as a central challenge for reliable personalized medical assistants. Resources are available at https://github.com/xiangjun2003/ObgynLongbench.