AI Content Moderation in Therapy Conversations
2026-05-25 • Human-Computer Interaction
Human-Computer InteractionArtificial IntelligenceComputation and LanguageComputers and SocietySocial and Information Networks
AI summaryⓘ
The authors studied how content rules in large language models (LLMs) like ChatGPT affect their ability to talk about sensitive topics in therapy sessions. They tested three popular moderation tools to see if these tools block parts of real therapy conversations. Their findings suggest that these safety systems might limit how well LLMs can function as therapists. This points out challenges for developers who want to use LLMs for emotional or formal therapy.
Large Language ModelsContent ModerationEmotional SupportTherapyChatGPTLlamaAlgorithm AuditOpenAI ModerationMeta Llama GuardGoogle Shield Gemma
Authors
Jiwon Kim, Claire Wang, Taeung Yoon, Sabelle Huang, Koustuv Saha
Abstract
Large language models (LLMs) are increasingly being used for emotional support. They are also being developed for formal therapy purposes. However, LLMs like ChaptGPT or Llama are often developed with content moderation guardrails that prevent them from discussing sensitive subjects with users for both liability and safety purposes, and this inability to broach these subjects may affect their capacity as therapists. In this study, we perform an algorithm audit on three state-of-the-art moderation systems (OpenAI's moderation endpoint, Meta's Llama Guard, and Google's Shield Gemma) to investigate the extent to which these systems flag the content of real-life therapy sessions as undesirable. Our results raise implications for the limitations that users and organizations may encounter when designing LLMs to play the part of a therapist.