Changing rating scales shifts large language model text scores
Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) are used to score texts on things like sentiment or toxicity, but these scores depend on the rating scale they use. The authors studied how changing the number of points or type of scale affects the scores LLMs give. They found that changing the scale design can noticeably shift scores and change how texts are categorized. This means scores from different studies using different scales may not be directly comparable. The authors suggest checking how robust results are across different scales for more reliable text analysis.
What this means in practice
- •For social media analysts: Adjust text scoring systems for platform content moderation to account for shifts caused by different rating scales in toxicity detection.
- •For political data teams: Design and interpret political leaning scores from LLMs more reliably by considering how scale changes influence normalized outputs.
Authors
Wonjeong Jo, Gyuwan Kim, Ming Shan Hee
Abstract
Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.