Clinical agents drop accuracy on noisy health records
EHR-RobustGym: Benchmarking and Training Agents for Robust Clinical Reasoning
Artificial IntelligenceMachine Learning
Summary
Hospital health records often have errors or missing details that make it hard for computer programs to answer medical questions correctly. The authors created a tool called EHR-RobustGym that tests how well these programs work when the health records are messy. They found that current large language models struggle to give correct answers when data is noisy, often performing much worse than on clean data. The authors also showed ways to train these programs to handle noisy records better, with improvements that apply to other health datasets too.
What this means in practice
- •For clinical ai developers: Test and improve clinical agents' ability to answer medical questions using noisy EHR data.
- •For healthcare data engineers: Use the benchmark to evaluate and enhance hospital data retrieval and query systems for robustness to record inconsistencies.
Authors
Yitong Qiao, Yancheng Jin, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu, Zhixuan Chu
Abstract
In hospital workflows, electronic health records (EHRs) are often noisy, and may not contain the evidence needed to confirm events or measurements referenced in a clinical query. Even when database retrieval succeeds, clinical agents can overlook such discrepancies and return plausible but unsupported answers. We introduce EHR-RobustGym, a scalable and interactive environment for evaluating and training robust clinical agents grounded in noisy EHRs. Built on MIMIC-IV hospital records (365K patients, 31 tables, and over 500M records), EHR-RobustGym comprises 5,486 Clean-Noise pairs spanning six clinical intents and both patient-level and population-level queries. The pairs test robustness to Record-level, Value-level, and Query-level noise, while interactive SQL/Python execution and outcome verification support trajectory collection and training. Evaluating multiple LLMs reveals substantial robustness gaps: average task success across proprietary and large-scale open-weight models drops from 62.2% on Clean questions to 37.9% on Noise questions. At k=4, pass^k consistency falls below 50% for most evaluated models, exposing instability in clinical task completion. Supervised fine-tuning and reinforcement learning in EHR-RobustGym improve performance, with gains generalizing to five external EHR benchmarks. Together, these results position EHR-RobustGym as a testbed for evaluating and improving the evidence-grounded robustness of clinical agents.