Aggregate comparisons reveal stable algorithm performance across health AI tasks
Rethinking How We Evaluate Methodological Progress in Health AI
Machine LearningArtificial Intelligence
Summary
It can be hard to tell which AI methods are best for analyzing health records because tasks and data vary a lot. The authors tested 12 different algorithms on two healthcare datasets using various tasks, some carefully created by experts and others randomly generated. They found that comparing methods overall works well across different tasks and datasets, meaning less effort might be needed to design specific evaluation tasks. Also, newer AI methods don't always outperform older ones like gradient-boosted trees. This suggests we should understand when certain methods work better for particular tasks instead of assuming the newest is best.
What this means in practice
- •For hospital data teams: Assess and select predictive models for clinical decision support using aggregate comparisons across diverse health record tasks without intensive task redesign.
- •For healthcare software developers: Incorporate gradient-boosted trees with wide and sparse EHR representations as reliable baseline models in clinical prediction software.
Authors
Florent Pollet, Matthew McDermott
Abstract
Methodological progress in artificial intelligence (AI) for electronic health records (EHRs) depends on our ability to determine which algorithms work better, and under which conditions. However, such progress is thought to be hindered by difficulties in reproducibility and in defining clinically meaningful evaluation tasks. We empirically study these barriers by re-implementing 12 historical and recent algorithms within a shared evaluation framework and evaluating them on two clinical datasets, MIMIC-IV and NWICU. We compare two complementary task families: expert-authored clinically meaningful tasks and generated tasks defined from randomly sampled event codes and prediction horizons. We ask whether relative algorithms comparisons transfer across task families and datasets, whether residual task heterogeneity contains useful methodological structure, and what a controlled comparison reveals about progress over the last decade. We find that aggregate pairwise comparisons transfer strongly across evaluation settings, including from randomly generated tasks to clinically meaningful tasks and across datasets. At the same time, clinically meaningful tasks exhibit greater task-method interaction, providing preliminary evidence that task properties can help explain when particular modeling choices are advantageous. Finally, newer algorithms do not consistently outperform earlier approaches: gradient-boosted trees remain highly competitive when paired with a modern, wide and sparse representation of the EHR. Together, these results suggest that useful methodological knowledge may require less task engineering than commonly assumed, while highlighting the importance of understanding the structured heterogeneity that remains across tasks and methods.