Medical language models struggle to use updated clinical knowledge well
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
Computation and LanguageMachine Learning
Summary
Medical knowledge changes all the time, and using old information can be dangerous. The authors created MedKIT, a tool to test if language models can not only remember new medical facts but also use them in different ways. They found that even the best models learn updates well but have trouble applying that knowledge in complex or new situations. This highlights a big challenge in making AI systems that safely and effectively use the latest medical information.
What this means in practice
- •For medical ai developers: Assess and improve how medical AI models incorporate and generalize new clinical evidence over time using MedKIT.
- •For healthcare software engineers: Evaluate updates in clinical decision support tools to ensure they safely handle evolving medical knowledge beyond mere fact recall.
Authors
Lukas Thede, Yash Kumar Atri, David Chen, Danielle Bitterman, Matthias Bethge, Tom Hartvigsen, Zeynep Akata
Abstract
Constantly evolving real-world knowledge necessitates models to be updated continuously. Especially in medicine, as clinical evidence changes over time, outdated knowledge can pose safety risks. Existing evaluations of knowledge integration focus on factual recall, offering limited insight into whether newly integrated knowledge is actually usable. Our benchmark MedKIT (Medical Knowledge Integration and Transfer) provides a granular evaluation of how models integrate and apply knowledge under realistic sequences of clinical updates. Each instance corresponds to a factual update derived from clinical evidence, paired with targeted probes that assess transfer across lexical variation, relational transformations, compositional reasoning, and open-ended operationalization, as well as locality tests for knowledge preservation. Using MedKIT, we conduct a large-scale empirical study of 12 knowledge integration strategies across 5 diverse models, including both general-purpose and medical LLMs. Our results reveal a consistent gap between recall and usable knowledge: while most methods achieve strong gains on the original update task and under lexical variation, relational generalization is limited, and no method yields meaningful improvements on compositional or operational tasks. These findings highlight a fundamental challenge in knowledge integration and position MedKIT as a testbed for developing methods that make newly integrated knowledge more consistently usable across tasks and contexts.