Large language models show no clear advantage using criteria extraction for depression severity
Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity
Computation and LanguageArtificial Intelligence
Summary
This paper compares two ways that large language models (LLMs) judge how severe depression is from social media posts: directly scoring how bad the depression sounds, or first identifying specific clinical signs (criteria) and then counting them. The authors find that using clinical criteria extraction doesn’t significantly improve accuracy compared to direct or chain-of-thought prompting, especially when thresholds for severity are preset. Sometimes, using criteria extraction even misses more severe cases. They also show that simpler models using dataset features perform similarly to more complex extraction methods.
What this means in practice
- •For mental health platform developers: Decide whether to integrate criteria extraction or direct severity rating from posts to monitor user depression symptoms, knowing criteria extraction offers no clear accuracy benefit.
- •For social media moderation teams: Improve detection workflows by understanding that using clinical criteria extraction may not reduce missed severe depression cases compared to direct language model scoring.
Authors
Xinkai Chen
Abstract
A large language model (LLM) can rate depression severity directly from a social media post or mark which clinical criteria the post shows and let code turn the count into a label. The latter is easier to audit because a clinician can check each marked criterion. We compare these approaches on two Reddit corpora using three LLMs (from 9B to frontier scale) and two questionnaires (PHQ-9, BDI-II), and measure agreement with quadratic weighted kappa. For the two frontier models, criteria extraction scores above chain-of-thought on one corpus only when its decision thresholds are fitted on labeled data. Neither model's gain is significant, with or without recalibrating chain-of-thought on the same labels. With thresholds fixed a priori from PHQ-9's criteria, extraction shows no gain on either corpus, even where models mark over two criteria per post. The 9B model behaves differently on a corpus from depression communities. It labels most posts severe, whether prompted directly or with chain-of-thought, while the a priori rule beats both without labels. After chain-of-thought is recalibrated on the same labels, no significant gap remains, consistent with a calibration effect. Yet higher ordinal agreement does not ensure better detection of severe cases. PHQ-9 criteria extraction misses most severe posts, and moving from direct prompting to chain-of-thought and then to extraction increases misses in nearly all comparisons. On the primary corpus, a relabeled stress dataset, a model using that dataset's own features, including word counts from the text, is not significantly different from frontier criteria extraction under the a priori rule.