Multimodal medical agent coordinates multiple specialist AI models effectively
MMedAgent-Plus: Scaling Multimodal Medical Agents across Modalities, Tasks and Workflows
Computation and LanguageArtificial Intelligence
Summary
Medical diagnosis often needs several different expert analyses, but AI models usually specialize in just one task at a time. The authors developed MMedAgent-Plus, which acts like a smart planner that can use many specialized AI tools across different medical tasks and data types. It chooses the right experts, runs them in order, and combines their answers into a clear and accurate response. Tests show it calls the right tools over 90% of the time and matches top specialists in giving correct final answers.
What this means in practice
- •For clinical ai developers: Integrate multiple specialized medical AI models into a single coordinated system for complex diagnostic workflows.
- •For healthcare it teams: Deploy a unified interface that manages different AI tools for various medical imaging and analysis tasks within hospital workflows.
Authors
Ruoxin Wang, Yuanting Pan, Binxu Li, Ruiyang Ji, Yaqian Chen, Jiayuan Ding, Qifan Wang, Lifu Huang, Mingjie Li, Yu Zhang, Ehsan Adeli, Kilian Pohl, Zhe Xu, Zihao Lin, Haoyu Dong, Yixin Wang
Abstract
Medical workflows often require multiple interdependent analyses, but existing specialist and generalist AI models are not designed to coordinate them. We introduce MMedAgent-Plus, a multimodal medical tool-use agent that combines a vision-language planner with 18 leading specialist models spanning 12 tasks and 8 modalities. Through a unified tool interface, the planner selects specialists, executes them in sequence, and synthesizes responses grounded in their outputs. The planner is instruction-tuned on 208,928 samples from 32 public medical benchmarks, covering single- and multi-tool interactions across single and multiple rounds. We separately evaluate tool orchestration, response synthesis, and end-to-end execution on held-out samples. MMedAgent-Plus achieves tool-call accuracies of 91.0%, 98.1%, and 80.7% in single-tool multi-round, multi-tool multi-round, and multi-tool single-round settings, respectively. Given correct tool outputs, MMedAgent-Plus achieves scores of 14.4-14.8 out of 15 for faithfulness, completeness, and conciseness across 600 evaluation samples. In end-to-end execution without pre-supplied tool outputs, MMedAgent-Plus correctly invokes the required specialist in 90.6% of 350 evaluation cases spanning seven tasks. Its final-answer accuracy reaches 0.98, matching the dedicated leading specialist and exceeding the strongest baseline (0.28).