Frozen AI models improve medical diagnosis by learning from new cases
Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI
Artificial Intelligence
Summary
Large AI models used in medicine often don’t learn after being deployed, missing new medical knowledge. The authors created a method that lets these models learn from new cases without changing their core design. Their approach uses external helpers to guide reasoning, remember trusted facts, and compare images, updating only when performance improves. This helps medical AI stay current and perform better on unseen cases.
What this means in practice
- •For medical ai developers: Implement AI systems that adapt to new clinical guidelines and cases without retraining the core model weights.
- •For visual reasoning system builders: Improve visual reasoning in deployed AI by integrating external visual knowledge bases that update from usage experience.
Authors
Yexiao He, Yucheng Tang, Pengfei Guo, Yufan He, Andriy Myronenko, Can Zhao, Ang Li, Daguang Xu, Dong Yang
Abstract
Large language models (LLMs) and vision-language models (VLMs) are usually frozen after deployment, so they do not learn from the cases they solve. This is especially concerning in medicine, where new clinical evidence, updated guidelines, and new therapies can change established practice. Fine-tuning can update the model, but it requires access to model weights and additional training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by saving experience only as text. To address these limitations, we present a model-agnostic framework that allows frozen LLMs and VLMs to learn from deployment experience through three forms of external expertise: a Skill that guides reasoning and tool use, a Knowledge Memory that stores reliable facts supported by earlier cases or trusted external evidence, and a Multimodal Knowledge Base that keeps visual examples and guides the model to relate each retrieved case to the current image. Instead of relying on a fixed validation set, a validation strategy keeps an update only if it helps on new cases without degrading performance on earlier ones. Across six benchmarks covering clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning, and with four open-weight and closed-source base models, our framework improves performance during online deployment by up to 34.2% over the base model on medical tasks, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.