M4Bench tests AI agents for clinical data extraction accuracy
M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents
Artificial Intelligence
Summary
Many medical studies rely on accurate data pulled from electronic health records, but defining how to get this data correctly is complicated. The authors introduce M4Bench, a set of tasks that checks if AI coding agents can properly recreate these data derivations and improve when using detailed instructions from clinicians. Their tests show that clear and complete instructions help AI perform better, while the format or generic familiarity helps less. Although AI improves with these instructions, perfectly matching expert outputs remains challenging. They provide open tools and data for others to evaluate AI in medical record processing.
What this means in practice
- •For hospital data teams: Improve clinical variable extraction from EHR data by using AI agents evaluated with M4Bench for accuracy and clinician-reviewed specifications.
- •For healthcare software developers: Develop AI-assisted tools for clinical data processing that leverage well-defined procedural specifications validated on diverse EHR datasets.
Authors
Hannes Ill, Rafi Al Attrach, Rajna Fani, Ahram Han, Ryohei Yamamoto, Marlene Louisa Moerig, Dukyong Yoon, Leo Anthony Celi
Abstract
Clinical research over electronic health records depends on derived variables whose definitions sit outside the schema, in item codes, time windows, missingness rules, and output-grain choices that change cohorts and downstream conclusions. We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU, to test whether AI coding agents can reproduce such derivations and how clinician-reviewed agent skills change their results. Its controls decompose the value of skills into specification completeness, delivery format, and schema familiarity. In a Docker-isolated Codex campaign (GPT 5.4 Mini and GPT 5.5), skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain: on an eight-task control subset, an operational spec alone raises reward by 0.162, and skills add only 0.024 on top, indistinguishable from zero. On a ten-task subset, decoy skills score like no skill and raw reference SQL nearly matches targeted skills, arguing against generic context or the skill format as the explanation. Schema obfuscation and restructuring preserve 86% and 70% of the skill gain (GPT 5.4 Mini, 26 MIMIC-IV tasks), a descriptive robustness check rather than a bound on memorization. A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent (+0.120, 21 of 28 tasks improving). These gains are partial agreement with the reference, not completed tasks: only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003). Skills are released as a versioned, clinician-reviewable markdown delivery format, with a family-level reference-validity audit. M4Bench releases tasks, skills, evaluation code, and sanitized per-run evidence as an open-source benchmark.