Automated medical image analysis with multi-agent evidence-based planning
A Multi-Agent Framework for Automated Machine Learning with Evidence-based Planning for Medical Image Analysis
Computer Vision and Pattern Recognition
Summary
Medical image analysis using deep learning is powerful but often requires expert tuning of model settings, which can be hard for doctors without coding skills. The authors present MedicalMLOmni, a system that uses multiple agents to automatically choose and optimize models by looking up published medical results and testing different options simultaneously. It works for many types of medical images and tasks, and performs better than other methods on most test problems. This tool helps medical experts use AI effectively without needing advanced programming knowledge.
What this means in practice
- •For medical imaging teams: Use MedicalMLOmni to automate model design and tuning across various medical imaging tasks without needing engineering experts.
- •For clinical software developers: Integrate a natural language interface to enable clinicians to configure and optimize imaging AI models easily for diverse diagnostics.$Commercial implications: Supports building user-friendly AI tools for healthcare providers, facilitating deployment of advanced imaging analysis in clinical settings.
Authors
Yiqing Shen, Mathias Unberath
Abstract
Deep learning has become the default tool for medical image analysis, driven by the open-source movement. However, achieving the best performance on a given task remains difficult, with architecture selection and hyperparameter optimization (HPO) still considered "more of an art than a science". Automated machine learning (AutoML) automates parts of model training, yet helps little in medical imaging: general-purpose approaches lack medical architectures and preprocessing, and medical-focused ones demand engineering expertise and remain confined to narrow tasks such as 3D segmentation. We introduce MedicalMLOmni, a multi-agent AutoML framework that automates model training for diverse single- and multi-modal medical tasks through a natural language interface. Specifically, MedicalMLOmni performs evidence-based planning by retrieving published results to guide model selection and configuration, and employs parallel HPO where multiple agents explore adjustment strategies concurrently. MedicalMLOmni achieves the best score on 19 of 20 benchmarks; PMC-VQA is the exception, where a vision-language model with broader pre-training leads. Backbone-matched controls attribute the gains to the framework, not the backbone. By removing coding requirements, MedicalMLOmni lowers the barrier for domain experts with rich datasets but limited engineering expertise.