Medical ml agents miss key safety checks despite good accuracy
Auto-MD-bench: Evaluating Autoresearch Agents for Real-World Medical Tasks
Machine LearningArtificial Intelligence
Summary
In medicine, it’s not enough for AI models to be just accurate; they must also give reliable risk estimates and work well across different patient groups. The authors built Auto-MD-bench, a set of real medical tasks, to test AI agents that create medical models. They found that while some agents make models with good scores, these models often give unreliable risk estimates and can perform poorly when used with different patients. This means just looking at accuracy scores does not show the full picture, and multiple measures are needed to judge medical AI models.
What this means in practice
- •For hospital data teams: Use Auto-MD-bench to evaluate medical AI models beyond accuracy by including risk calibration and robustness across patient groups.
- •For clinical software developers: Incorporate multiple evaluation metrics from Auto-MD-bench into AI model development pipelines to ensure safer deployment in healthcare settings.
Authors
Shenqi Zhou, Qiran Zou, Wenhao Zhao, Zhenyu Liao, Dianbo Liu
Abstract
Autonomous agents are increasingly capable of developing machine learning (ML) models end to end. In medical practice, however, strong performance on the target metric does not fully characterize how good the models are in real-world use cases. For example, medical ML models should also provide well-calibrated risk estimates and remain robust across patient populations. Yet autonomous model development is typically guided by task objectives that emphasize simple predictive discrimination (e.g. AUROC). This raises a key question: are these objectives sufficient to guide agents toward models that meet the broader requirements of real-world medical use? In this study, we introduce Auto-MD-bench, a benchmark of 15 real-world medical tasks from 10 clinical datasets, spanning survival analysis, classification and sequential prediction, with external validation across cohorts, hospitals and time periods. We evaluate four agents: AIDE, AI Scientist v2, Codex, and Autoresearch. The strongest agents achieve competitive primary scores, yet their predicted risks are less calibrated than those of the simple baseline models, which could lead to severe medical accidents. Moreover, models with nearly identical primary scores can exhibit markedly different calibration, while performance varies substantially across population shifts. These findings show that primary task performance alone is insufficient to assess autonomous clinical modeling and multiple different metrics widely used in healthcare practice need to be taken into consideration.