Synthetic hospital creates realistic open benchmark for clinical AI
Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark
Artificial IntelligenceDatabases
Summary
Access to real medical records is tightly restricted due to privacy, making it hard to build and test AI tools for healthcare. The authors created Synthetic Hospital, a fully artificial and open medical record system based on public educational sources, with accurate medical facts and timelines. It mimics real hospital data and was hard for doctors to distinguish from actual patient charts. Tests show current AI models still struggle with these records, showing Synthetic Hospital is a realistic way to evaluate and improve clinical AI.
What this means in practice
- •For clinical ai developers: Train and test medical AI models on fully synthetic but realistic patient records without privacy concerns.
- •For medical software engineers: Build and validate EHR-integrated tools using a synthetic hospital environment that mimics real healthcare data and infrastructure.
Authors
Christine Park, Valerie Chen, Tim Dettmers
Abstract
Frontier language models are rarely used in clinical workflows because the realistic, longitudinal benchmarks needed to develop them are scarce. Real electronic health record (EHR) data cannot be openly shared due to privacy, ethics or data use issues and it does not contain verifiable ground truth since the chart records only reflect what clinicians documented. We introduce Synthetic Hospital, an open, fully synthetic, fact-grounded longitudinal EHR benchmark that resolves the open sharing and verifiable ground truth barriers. Built entirely from public medical-education material with no protected health information, it comprises 1,268 longitudinal patients and 5,602 encounters, where every diagnosis, finding, and temporal relation is grounded in standard ontologies (ICD-10-CM, SNOMED CT, LOINC) and with a complete provenance chain back to its source medical education material. Synthetic Hospital is served through a simulated hospital record system that mirrors real EHR infrastructure (standard interoperability APIs, role-based access and function-calling interface). In a blinded review, physicians distinguished its records from real patient charts at near-chance rates (53\%). Across 10 frontier and open models, none approaches ceiling: the best model reconstructs a patient's longitudinal problem list with a severity-weighted F1 of 0.73, level with the mean of seven physicians on a matched subset but well below the best of them (0.89), and misses roughly half of clinically relevant findings when summarizing a chart. Overall, these results highlight that Synthetic Hospital is a difficult and realistic test of clinical AI performance.