Large language models struggle to match human politeness judgments
Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms
Computation and Language
Summary
This paper looks at how well large language models (LLMs) understand politeness compared to humans. The authors tested seven different LLMs on two sets of politeness judgments and found that the models often agreed with each other more than they agreed with people. Models tended to label statements as neutral more than humans do and missed some impolite cues. The study suggests we need better ways to check if AI understands social rules like politeness the way humans do.
What this means in practice
- •For conversational ai developers: Improve politeness detection in chatbots by identifying where models systematically misjudge polite or impolite language.
- •For customer support teams: Assess and refine automated messaging tools to better reflect human perceptions of politeness and reduce communication errors.
Authors
Rong Wang, Kun Sun, Yadong Guo
Abstract
Despite strong performance on standard benchmarks, it remains unclear whether large language models (LLMs) evaluate social pragmatics in ways that align with human judgments. We evaluate LLM politeness judgments using two English-language datasets with complementary annotation formats: continuous human ratings and three-way categorical labels. Across the seven evaluated models, we find that inter-model agreement is stronger than model--human agreement. Strategy-level analyses suggest that model--human alignment is associated with explicit linguistic cues, while some rapport-building strategies occur more frequently in misaligned cases. In the categorical task, model predictions exhibit systematic neutral compression, characterized by the overproduction of Neutral labels and the underprediction of Impolite labels. This pattern persists when expert consensus is used as the reference on a diagnostic subset. Our findings highlight the need for pragmatic evaluations that go beyond aggregate agreement metrics by examining directional patterns of model--human disagreement across different human references.