Large language models show hidden failures in content analysis tasks
High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis
Computation and LanguageComputers and Society
Summary
Large language models (LLMs) are often used to automatically label text based on predefined codes, which helps social scientists analyze text data. The authors found that even when these models seem accurate overall, they often make inconsistent or unreliable labeling decisions that go unnoticed by standard accuracy checks. These problems include instability when the model runs multiple times, sensitivity to changes in text or context, and varying meanings of labels. The authors show that without careful additional testing, these LLM-based tools should not be fully trusted to produce stable, valid labels.
What this means in practice
- •For computational social science teams: Improve validation procedures for LLM-based text coding pipelines to ensure more reliable and stable content annotations.
- •For natural language processing engineers: Develop robust methods for detecting and mitigating instability and domain sensitivity in LLM-driven text classification.
Authors
Elena Senger, Yuri Campbell, Rob van der Goot, Barbara Plank
Abstract
Large language models (LLMs) are increasingly used for codebook-based content analysis in computational social science, where codebooks translate theoretical concepts into labeling schemes. While prior work reports strong predictive performance, it remains unclear whether these systems produce reliable and robust text annotations. We conduct a large-scale empirical failure-mode analysis of LLM-based coding pipelines across four datasets and five models by systematically evaluating their stability, robustness, and conceptual validity under controlled perturbations, domain shifts, and optimization procedures. We uncover multiple failure modes that are largely invisible under standard metrics, showing that high aggregate scores frequently coexist with severe label-level disparities, instability across runs, domain sensitivity, and shifts in label interpretation. Importantly, these failure modes persist under widely used optimization strategies and remain largely invisible under standard metrics such as accuracy and micro-F1. Our results demonstrate that current LLM-based coding pipelines cannot be treated as stable measurement tools without targeted validation beyond aggregate performance.