Ai coaching requires layered human oversight for mental health safety
Scalable Oversight for AI in Mental Health: Lessons from 350,000 AI Coaching Conversations between Therapy Sessions
Computers and Society
Summary
AI tools can help people between therapy sessions, but checking every AI message by clinicians is too hard and can actually lead to mistakes. The authors studied over 350,000 AI coaching chats and created a system with three layers: design that prevents problems, real-time checks, and ongoing doctor reviews. They improved the AI by learning from these reviews and give advice for mental health workers who use AI. This approach balances safety with scale better than just watching everything closely.
What this means in practice
- •For mental health professionals: Use a three-layer oversight framework to safely integrate AI coaching tools between therapy sessions.
- •For digital health app developers: Design AI mental health chatbots with preventive, monitoring, and evaluation layers informed by clinical review insights.
Authors
Matthew A. Scult, John L. Havlik, Kevin Ramotar, Ethan Goh, Manoj Kanagaraj
Abstract
Clinician review of every AI output is often proposed as a safeguard in mental healthcare, but vigilance research suggests this approach fails at scale and may paradoxically reduce safety. Drawing on our experience deploying an AI coaching tool across 350,000+ conversations between therapy sessions, we describe how we arrived at a three-layer human-on-the-loop oversight framework combining preventive design, real-time monitoring, and continuous clinician evaluation. We show how specific findings from clinical review drove iterative improvements, and offer practical recommendations for mental health professionals evaluating AI systems.