Reasoning boosts AI story depth mostly by AI judges not humans
Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging
Machine LearningComputation and Language
Summary
This paper looks at whether large language models (like GPT-5 and others) produce stories with more psychological depth when they use reasoning. It turns out that human readers don’t strongly agree that one model is better than another in this way, with preferences varying a lot. On the other hand, AI-based judges consistently favor the stories with reasoning and tend to focus on things like sentence length. The authors show that AI judges trained on one set of data may not reliably reflect human judgments when the stories are different or more subjective.
What this means in practice
- •For ai system evaluators: Improve evaluation methods by accounting for variability and subjectivity in human preferences beyond AI judge scores.
- •For creative ai developers: Design story generation systems with awareness that reasoning steps may not reliably enhance perceived psychological depth to human readers.
Authors
Ruichen Zheng, Yihe Wang, Fabrice Y Harel-Canada, Sara Khosravi, Zeynep Senahan Yildiz, Amit Sahai, Nanyun Peng
Abstract
LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set may not guarantee valid measurement when outputs are closely matched and human preferences are subjective. We study this failure mode through psychological depth in short stories. Seven human readers and an LLM-judge ensemble selected on the original scalar Psychological Depth Scale dataset ($ρ= 0.646$) evaluated 60 blinded, prompt-matched story pairs from GPT-5 vs.\ GPT-4o and DeepSeek-R1 vs.\ DeepSeek-V3. Human preferences showed no universal reasoning advantage: GPT-5 was modestly preferred over GPT-4o (60.0--62.9\%), whereas DeepSeek-R1 trailed V3 (42.9\%), and inter-reader agreement was near chance (Krippendorff's $α= 0.070$), with within-reader consistency and recurring weighting patterns suggesting structured heterogeneity rather than random responding. The judge, by contrast, favored reasoning outputs in 89.0\% of dimension-level comparisons and 59 of 60 pairs on aggregate PDS, uniformly across all five evaluator configurations, and its scores were associated with surface features such as sentence length and lexical diversity. These results suggest that development-set performance is insufficient evidence for deployment validity on a shifted distribution, and that point-estimate judges can obscure the heterogeneity in subjective human evaluation.