Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts
Computation and Language
Summary
The authors studied how well large language models (LLMs) detect psychological distress in online posts from different identity-based communities. They found that humans from the same community rated distress differently than outsiders, showing that context matters. Most LLMs tended to overestimate distress, especially in posts the community saw as mild or not distressing. This suggests that current LLMs may not fairly represent the views of the communities they evaluate, which is important for mental health applications. The authors highlight the need for more context-aware AI in this area.
Authors
Andrew Aquilina, Xiang Lorraine Li, Yu-Ru Li
Abstract
Judgments about psychological distress are socially situated: what counts as concerning hinges on community norms around emotional expression, vulnerability, and help-seeking. Yet large language models (LLMs) used for distress detection are typically aligned to a single, undifferentiated standard. How well do these models capture the perspectives of the communities whose language they assess? We address this question through a perspectivist annotation study in which 321 participants provided 9,587 judgments on 1,198 Reddit posts spanning six identity-based communities, yielding community-specific labels. Raters in the contextualized in-group condition show a modest tendency to agree more with their community than uncontextualized out-group raters (OR = 1.18), an effect varying significantly across communities. We then evaluate nine open-weight LLM configurations and four frontier configurations against these labels. Open-weight LLMs systematically over-estimate distress: when communities perceive none-to-mild distress, these models achieve only 31-44% accuracy, predominantly producing false positives. GPT-5 and Gemini 2.5 Pro show the same none-to-mild inflation even when their full-sample over/under rates are mixed, while Claude Opus 4 is more conservative. This pattern does not simply mirror an outsider reading position: uncontextualized out-group human aggregates were nearly symmetric, with 18% over-estimation versus 19% under-estimation. Instead, the models that inflate none-to-mild cases exhibit a distress prior that exceeds both contextualized in-group and uncontextualized out-group human judgments. These findings have implications for equitable AI deployment in mental health contexts, where miscalibrated distress detection may unevenly affect the communities being assessed.