Llms balance refusal and response risk when addressing suicidal thoughts
Risk and Refusal: Mechanisms of LLM Responses to Suicidal Ideation
Human-Computer InteractionComputation and LanguageComputers and SocietySocial and Information Networks
Summary
Sometimes large language models (LLMs) are programmed to refuse to respond to dangerous or harmful requests, including when users mention suicidal thoughts. The authors studied how eight different LLMs react when users share suicidal ideas, finding that refusing to engage can sometimes make things worse instead of safer. They found that providing a special, clinician-designed response protocol helped models respond more safely and engage better. This research highlights the need for LLMs to refuse only when truly necessary and to reply carefully to sensitive disclosures like suicide.
What this means in practice
- •For llm safety teams: Improve model refusal policies to safely engage with users disclosing suicidal thoughts while reducing harmful response risks.
- •For mental health chatbot developers: Integrate clinician-developed response protocols into chatbots to lower risks in replies to users expressing suicidal ideation.
Authors
Drishti Goel, Soorya Ram Shimgekar, Jeongah Lee, Violeta J. Rodriguez, Daniel S. Brown, Ravi Karkar, Koustuv Saha
Abstract
Refusal is a common safety mechanism in large language models (LLMs), yet in contexts of suicidal ideation (SI), refusing to engage can itself constitute a safety failure. We evaluate how eight instruction-tuned LLMs respond to SI and self-harm disclosures across three datasets, varying whether models are informed that a disclosure expresses SI and whether they receive a clinician-developed response protocol, and complement this with mechanistic analyses in six open-weight models. Our results reveal a nuanced relationship between refusal and response-level risk. At baseline, elevated risk in model responses is driven primarily by inattention to the disclosure, alongside refusal to engage. Mechanistic analyses suggest that models may refuse SI disclosures through the same direction that mediates their refusals of harmful requests. However, informing models of the SI context, or suppressing refusal directly, increases engagement without reliably lowering response risk, and may raise it. In contrast, the clinician-guided protocol consistently reduces response-level risk while also reducing refusal. Together, these analyses and ablations motivate the need for models that refuse only when warranted and engage in interactionally safe ways when users disclose SI.