Medical AI systems resist prompt attacks with judgment checks
The Model Knows Better: Defending Medical Multi-Agent Systems Against Prompt Injection
Computation and Language
Summary
Medical AI systems that help doctors can be tricked by attackers who add misleading instructions to their input. The authors found that teams of AI models are not always safe from such attacks. They introduced a new method that works like a fact-checker by making the AI verify its own medical decisions against the patient data, reducing errors significantly. This method does not require extra training and works well on different AI models and challenges.
What this means in practice
- •For medical ai developers: Integrate judgment verification steps to protect AI diagnostic tools from misleading input attacks without retraining models.
- •For healthcare it teams: Deploy enhanced defenses in clinical decision support systems to maintain accuracy even when input data might be manipulated.
Authors
Fardin Ahsan Sakib, Ziwei Zhu, Ozlem Uzuner
Abstract
Medical multi-agent systems, in which LLM agents discuss a case, are increasingly studied for clinical decision support. Prompt injection, text that an attacker adds to a model's input, can manipulate LLMs into wrong decisions. How it affects these systems, and how to protect them, remains unclear. Across two systems, two model families and three benchmarks, a single injected passage succeeds in up to 61% of cases, and collaboration does not consistently protect against it. Existing defenses help only partly: they miss injections that look like ordinary clinical text. Yet when asked to check the injected text against the patient's findings, a model that followed the injection usually rejects it. We therefore propose a training-free defense that verifies the clinical judgments behind the system's decision. Following clinical practice, it separates the patient's findings from clinical judgments, such as diagnoses, recommendations and instructions, and checks each judgment against the findings alone, so the decision rests only on supported judgments. Our defense outperforms all baselines on every benchmark, lowering attack success on MedQA to 4-7% while maintaining accuracy on cases without an attack, and remains effective with a medical model and against defense-aware and adaptive attacks.