Memorisation bias affects medical AI diagnostic accuracy on returning patients
Memorisation bias in medical AI
Machine LearningComputers and Society
Summary
Medical AI models sometimes remember details from patients' past medical records they were trained on. This can change how the models predict a patient's future health, even years later. The authors found that when a patient returns with new conditions not in their past data, the AI can miss diagnoses, but if their health is unchanged, the AI tends to be overly confident. This problem, called memorisation bias, may create risks when AI tools are used for patients who contributed data before. The authors suggest changes in how models are trained and used to reduce such risks.
What this means in practice
- •For hospital data teams: Detect and manage the impact of memorisation bias in AI models assessing patients who contributed historical data.
- •For healthcare ai developers: Design AI training and deployment protocols that mitigate risks from memorisation bias affecting future patient predictions.
Authors
Moritz A. Knolle, Martin J. Menten, Laurin Lux, Mélanie Roschewitz, Emma A. M. Stanley, Georgios Kaissis, Daniel Rueckert, Ben Glocker
Abstract
Medical AI models hold immense potential to improve patient outcomes, but they are also known to unintentionally memorise individual records from their training datasets. While such memorisation has been linked to targeted privacy attacks, its consequences for clinical deployment, where patients may be assessed by a model that saw their historical data during training, remain poorly understood. Here we show that predictions on a patient's unseen future data can change significantly if a model observed that same patient's anonymised historical data during training, a phenomenon we term "memorisation bias". We demonstrate that this bias exists across diverse data modalities and model architectures, and over prolonged time spans: in some cases, memorisation bias persists on future records acquired decades after the historical records used for training. Moreover, in simulated prospective deployment, memorisation bias has asymmetric effects on the diagnostic accuracy of returning data contributors. When a patient returned with a de novo condition absent from their historical records in the training dataset, diagnostic sensitivity decreased significantly compared to an otherwise identical model not trained on their historical data. Conversely, when their health state was unchanged, both sensitivity and specificity were significantly inflated. Our findings reveal a previously uncharacterised risk in medical AI that arises when a model is deployed on patients who contributed to its training data. This exposes a shortcoming of current model development practice: the de-identification measures designed to protect patients' privacy make it difficult to identify returning contributors and exclude them from the AI-assisted interpretation of their own future data. Mitigating memorisation risks may thus require changes to current model training and deployment protocols.