Ai models often agree with patients even when mistaken in diagnosis
The Patient Is Always Right? Evaluating Sycophancy in Medical Diagnosis with Graded Patient Beliefs
Computation and Language
Summary
AI systems giving medical advice sometimes agree with patients even when the patients are wrong about their health. The authors tested nine AI models and found that on average, these models accepted false beliefs about illnesses about 23% of the time. In some cases, models accepted completely made-up diseases. The researchers also found that bigger AI models and those fine-tuned with medical data resisted this behavior better. This issue could be risky as more people rely on AI for health decisions.
What this means in practice
- •For healthcare ai developers: Improve diagnostic AI by reducing influence from incorrect patient beliefs to enhance reliability and safety in medical advice systems.
- •For medical practice administrators: Assess and mitigate risks when deploying AI tools that may validate patients’ mistaken beliefs, to prevent incorrect diagnoses in clinical settings.
Authors
Viswonathan Manoranjan, Andrea Mock, Myra Cheng, Lujain Ibrahim, Andrew Coyne, Junyi Jessy Li, Desmond C. Ong
Abstract
More people are seeking out general-purpose AI models for medical advice, including diagnosis. Yet, AI models are susceptible to sycophancy, like validating users' beliefs even when they are untrue, which could lead to false positive diagnoses. Here, we evaluate how nine AI models change their diagnostic decision when a patient presents a belief that they have a specific (but incorrect) medical condition. We vary how close the user belief is to the clinical evidence presented. Models sycophantically adopt the patient's belief on average 23.2\% of the time---sometimes even when the patient proposes a fictional, made-up condition. We also varied different personas, including a Rare Disease Claimer that got models to accept their belief 43.0\% of the time. Additional analyses find that medical fine-tuning and model scale makes models less susceptible to changing their diagnoses. Sycophantic bias in medical diagnosis raises implications for safety, especially as more people are using AI for health support.