Large language models improve peer mental health support with varied success
Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generation
Computation and LanguageArtificial IntelligenceComputers and Society
Summary
Many people seek mental health advice from peers on websites like Reddit, but many questions get no replies. The authors created a new dataset from these online communities and trained language models to generate supportive responses that better match the ways people actually talk and cope. They found training helps the models give more aligned, helpful answers but not equally across all subgroups or mental health topics. The models tended to focus more on problem-solving advice over emotional support after training. This research shows the potential and limits of using AI to support online mental health conversations.
What this means in practice
- •For mental health app developers: Create AI tools that generate community-aligned support responses tailored to specific mental health forums using the COPES dataset.
- •For online community moderators: Deploy fine-tuned language models to assist in responding to unanswered mental health queries with peer-experience-informed advice.
Authors
Mohit Chandra, Nabin Kim, Eli Min, Aamogh Sawant, Tanmay Sutar, Munmun De Choudhury
Abstract
As access to professional mental healthcare remains limited, many individuals turn to online platforms such as Reddit to seek peer support situated within human lived experience. However, a significant portion of such queries go unanswered, presenting an opportunity for using Large Language Models (LLMs) to fill this gap. While LLMs have demonstrated strong performance on clinical benchmarks, their ability to generate lived-experience informed and community-aligned peer support is underexplored. Addressing this gap, we introduce the COmmunity-centered Peer Engaged Support (COPES) dataset and a three-axis evaluation framework to assess LLM alignment with community perspectives to mental health support seeking queries. Evaluating zero-shot and post-trained (SFT and DPO) models, we show that post-training on COPES significantly improves Strategy Alignment (>50% for general-purpose models) and alignment in Emotion & Tone. However, we also observe that such improvements are heterogeneous and alignment improvements vary significantly across subreddits and requested coping strategies. Furthermore, post-training induces distributional shifts, heavily favoring problem-focused recommendations while suppressing emotion-focused strategies. Together, this work shows that while curating community-driven data improves the alignment of LLM responses, model performance remains disparate across distinct sub-communities and specific mental health needs.