THReD system reliably turns electronic health records into AI datasets
Benchmark of THReD: Evaluating an LLM Agent and Its Preprocessing Tools for Turning Electronic Health Records into AI-Ready Datasets
Artificial Intelligence
Summary
Electronic health records (EHRs) are complex and messy, which makes it hard for AI to use the data. The authors evaluated a web-based system called THReD that uses a large language model chat agent to help turn EHR data into clean, AI-friendly datasets. The system performed well on many tasks, including catching errors and handling tricky situations, and it produced consistent, accurate results on both synthetic and real patient data. This means THReD could help hospitals and researchers prepare health data for AI analysis more reliably.
What this means in practice
- •For hospital data teams: Prepare electronic health record extracts into accurate, ready-to-use datasets for AI modeling with minimal human intervention and error.
- •For health it developers: Integrate LLM-based agents and data profiling into tools that clean and label clinical data for research or operational use.
Authors
Javad Mohammad Alizadeh, Huanmei Wu
Abstract
LLM agents are increasingly built into data-analysis software, yet little evidence shows how reliably they preprocess clinical data or whether the tools they invoke give correct, reproducible output. We report a benchmark of THReD, a web-based system that turns electronic health record (EHR) extracts into AI-ready datasets through a chat agent whose proposed steps run only after user confirmation. The agent benchmark has 141 tasks (100 templates) in seven categories, including goal-level requests, failure recovery, requests calling for no action, schema variation and prompt injection through the data, on synthetic tables calibrated to a regional health information exchange; success is judged by checks on the produced data. Over 2,550 episodes with Claude Haiku 5.5 in four context configurations, the deployed agent completed 92.7% (95% CI 88.1-96.4) of tasks on its first answer, abstained on every advice request, recovered from every injected fault, and never proposed an action requested by planted text. Without the data profile, completion fell to 58.0%; goal-level requests were hardest (80.6%). Six experiments on real records of two chronic-disease cohorts and on synthetic data at scale validate the tools: 91-92% of inferred patient classes and 99.7-99.8% of inferred vital-sign labels were correct in masked reconstruction; a dictionary builder combining curated dictionaries with LLM suggestions mapped 99.0% of 2,208 raw variants correctly; all 2,176 replays of 128 saved pipelines gave byte-identical files under pinned dependencies; subgroup differences in inference accuracy vanished within facility and lookup key; and the same-day consolidation rule changed 90-day emergency revisit rates by up to 3.2 points without shifting cohort composition by more than one point. Processing time grew roughly linearly to 5.5 million records while the prompt stayed at about 13 thousand tokens.