Large language models struggle to update advice with new medical knowledge
Large language models show systematic failures in incorporating evolving medical knowledge
Artificial Intelligence
Summary
Medical rules for doctors change over time, but big language models that help with medical advice often fail to update their answers fully to match the newest rules. The authors created a test called ChronoEvo to check if models notice and use new medical guidelines correctly. They found that while models usually recognize changes and often give correct answers, they only fully update their recommendations about 60% of the time. This means these models can keep giving outdated advice even when they know the rules have changed.
What this means in practice
- •For clinical ai developers: Design testing methods to better identify when AI models fail to fully update medical advice based on changing guidelines.
- •For healthcare technology teams: Improve medical decision support systems by addressing gaps in how AI models incorporate evolving clinical recommendations.
Authors
Min Zeng, Zaifu Zhan, Kai Yu, Rui Zhang
Abstract
Medical knowledge evolves as clinical guidelines revise recommendations, yet whether large language models (LLMs) faithfully incorporate such changes remains unclear. We introduce ChronoEvo, comprising 1,709 recommendation-level revisions across 12 forms of medical knowledge change, to distinguish recognition of revised guidance from incorporation into clinical recommendations. In the primary model, 90.8% of responses recognized revisions and 80.2% were clinically correct, but only 58.5% fully incorporated guidance, yielding a 32.3-percentage-point recognition-incorporation gap. This dissociation persisted across 14 models and was corroborated by two clinicians' adjudicated assessment of 300 responses, yielding a 40.3-percentage-point gap in the validation sample. Changes in recommendation strength and patient eligibility were associated with retention of outdated guidance. A revision-salience intervention reduced outdated-content retention and improved clinical correctness but did not significantly improve full incorporation. These findings identify faithful incorporation of evolving medical knowledge as a distinct challenge for clinical LLM reliability, even when updated guideline evidence is available.