Health systems need to check error overlap between clinicians and AI models
Complementary Vulnerabilities: Auditing and Governing Clinician-LLM Error Overlap
Computers and Society
Summary
Sometimes doctors and AI language models make mistakes, but those mistakes might be different because of how they think and work. This paper says hospitals should test how often doctors’ errors match AI errors before using these tools. If the mistakes overlap too much, the AI doesn’t add safety by catching doctors’ errors. The authors suggest ways to keep errors different over time so AI can help clinicians better.
What this means in practice
- •For hospital data teams: Measure how often clinician and AI diagnostic errors overlap in real cases, to decide if AI tools improve safety.
- •For clinical ai deployment teams: Implement governance processes that keep clinician and AI errors independent over time to maintain complementary decision support.
A position paper. It proposes an approach and reports no results.
Authors
Mohsen Amoei, Kathleen Fraser, Dan Poenaru
Abstract
Health systems are deploying large language models (LLMs) into clinical workflows, often on the assumption that a second automated check can only reduce error. But a second reasoner reduces error only insofar as it fails differently from the first. We say that a clinician and a model have complementary vulnerabilities to the extent that they break down through different mechanisms and on different cases. The two reasoners' failure substrates differ: they are embodied and cognitive in the clinician, and statistical and generative in the model - so their weak points need not line up. That divergence is plausible rather than guaranteed, and the independence it creates is fragile, eroding when clinicians defer to the model or the model conforms to the clinician. We therefore propose a predeployment audit in which each site measures error overlap on its own cases. We also propose a governance model with which the site keeps those errors independent over time. Health systems should then judge a clinical artificial intelligence (AI) tool by whether its failures overlap with their clinicians' failures, as well as by its accuracy.