Simulation reveals safety risks in medical triage chatbots
SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots
Computation and Language
Summary
Medical chatbots are used to guide patients, but current tests often miss how real conversations happen, especially in different languages and with confusing or emotional users. The authors created a way to simulate real user interactions to better find when chatbots make dangerous mistakes. They tested chatbots used in India, South Africa, and Kenya and found that chatbots sometimes wrongly downplay serious symptoms if a user minimizes them. This method helps spot risky responses before chatbots are widely used in healthcare.
What this means in practice
- •For healthcare ai teams: Use multi-turn user simulations to test triage chatbots for realistic and culturally relevant safety risks before deployment.
- •For multilingual chatbot developers: Improve chatbot robustness by identifying failure modes in regional dialects and complex user behaviors during health conversations.
Authors
Deepali Mittal, Dheeraj Mundhra, Yulia Sidorova, Tersia de Kock, Oli Munns, Sofie S Meyer
Abstract
LLM-based chat agents are increasingly used for complex, multi-turn, multilingual medical triage in low- and middle-income countries (LMICs), yet evaluation methods focus on single-turn, English-only benchmarks that primarily test the chatbots' knowledge. In practice, users are often confused, overwhelmed, or emotionally vulnerable, giving unclear, misspelled single-word answers or long messages in regional dialects, and the context of a conversation can shift rapidly. We propose SIM-UT (Simulation of User Behaviour for Triage), a multi-turn, simulation-based evaluation method that identifies ecologically valid, contextually relevant failure modes of LLM-based triage chat agents. Using SIM-UT, we evaluated three real-world WhatsApp-based health chatbots: a respiratory screening programme in India, a hearing health support programme in South Africa, and a diabetes and lifestyle coaching agent in Kenya. We compared a medically fine-tuned open model (MedGemma) against general-purpose frontier models (ChatGPT and Claude) across 5,000+ multilingual simulated conversation turns grounded in the personas, constraints, and communication patterns of the real user populations, over 2-4 iterative rounds. A key failure mode emerged in both model families: de-escalation risk. The agents initially identified medical emergencies correctly, but when the user minimised the symptoms without introducing new clinical information, they de-escalated, exhibiting clinically dangerous sycophancy, for example agreeing that vaginal bleeding in pregnancy was "not a problem" or that decreased fetal movements at 32 weeks "can be normal". We recommend user simulation-based behavioural evaluation, grounded in real user context and covering adversarial and edge cases, as standard practice for triage chatbots in production, so that safeguards can be added before deployment or in subsequent iterations.