Llm benchmark measures safety in high-risk mental health talks
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
People are using large language models (LLMs) to get help with mental health, but it’s unclear how safe these models are when conversations get serious. To address this, the authors created K-Bench, a test tool checked by doctors that evaluates how well different AI models handle tough mental health topics like suicide and self-harm. They found some models give good supportive answers, while others vary a lot in spotting risk. The benchmark also includes a public leaderboard so developers can see how their models compare over time.
What this means in practice
- •For mental health app developers: Evaluate and improve AI chatbots’ safety and support quality in sensitive mental health conversations using K-Bench.
- •For clinical safety teams: Use K-Bench to verify that deployed LLMs in healthcare settings manage suicide and self-harm dialogs safely and consistently.
Authors
Laura M. Vowels, Matthew J. Vowels, Shivali Sharma, Apoorv Jha, Rehnuma Choudhury, Wasseem El Sarraj, Rachel Francois-Walcott, Aruba Hussain, Sarah Ingram, Angela Loulopoulou, Adva Segal, Elena Volkova
Abstract
% !TEX root = ../main.tex People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk conversations remains poorly characterised. We developed K-Bench, a clinician-calibrated, protected benchmark evaluating 125 model configurations representing 33 base models from 14 providers across a fixed cohort of 200 multi-turn vignettes involving suicide, self-harm, domestic violence, substance misuse, and no-risk presentations. Synthetic patient conversations showed substantial distributional overlap with real human-AI conversations. A frozen GPT-4o judge achieved 94.2% exact agreement with clinician consensus across 6,751 eligible item comparisons from 151 clinician-rated transcripts. Leading models combined strong supportive conversation with combined-risk scores above 95, whereas risk exploration exposed substantial variation among lower-performing configurations. Therapeutic prompting produced configuration-specific gains concentrated among weaker models, while elevated reasoning produced no average improvement. K-Bench combines broader clinical coverage and configuration-scale comparison with a continuously updated public leaderboard whose operational test materials are protected from direct optimisation. The leaderboard is available at www.k-bench.ai.