Large language models rate summary quality differently than humans do
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
Computation and Language
Summary
People often use large language models (LLMs) to judge the quality of summaries by comparing their scores to human scores. The authors show that even when LLMs and humans mostly agree on overall quality, they don't find the same summary examples equally hard to judge. They studied these differences by breaking down judging difficulty and found that LLMs and humans struggle with different aspects of summaries. This mismatch depends on the evaluation dimension, like coherence or consistency, and helps explain limits in current evaluation methods.
What this means in practice
- •For nlp system developers: Improve summary evaluation systems by targeting where human and LLM judgment difficulties differ to create more reliable automatic judges.
- •For content moderation teams: Use insights on dimension-specific judging difficulty to better combine human and LLM assessments for evaluating text quality.
Authors
Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne
Abstract
Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs find the same evaluation cases difficult. In this paper, we study this problem in summarization evaluation from a psychometric perspective. We fit Many-Facet Rasch Models separately to human and LLM ratings to decompose scores into latent summary quality, rater severity, dimension severity, and rating-scale thresholds. Building on this decomposition, we define residual hardness as a model-adjusted measure of judging difficulty and compare whether human and LLM judges share the same hardness structure. Across 17 open-weight LLM judges on SummEval, we find that moderate alignment in latent summary quality does not imply alignment in residual hardness. Human and LLM judges differ in which summary--dimension units remain difficult, and this mismatch is strongly dimension-dependent. Consistency shows a pronounced LLM-hard shift, whereas coherence shows a human-hard shift. We further show that human-easy but LLM-hard cases are partially predictable from observable source--summary properties. These findings suggest that aggregate human alignment reflects only part of LLM-as-a-judge reliability, while psychometric residual diagnostics support more informative judge evaluation and more targeted human--LLM collaboration.