Large language models help experts quickly improve text labeling guides
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Computation and LanguageArtificial IntelligenceHuman-Computer InteractionMachine Learning
Summary
Labeling millions of documents with expert insight takes a long time, especially since guidelines called codebooks need months to prepare. The authors show that by using multiple large language models (LLMs) to find where they disagree on labels, experts can focus their attention more effectively. They tested three ways experts can fix these disagreements and found that having experts provide detailed explanations (rationales) for disagreements improved labeling accuracy the most. This approach can reduce the time needed to refine labeling guides from months to days without losing quality.
What this means in practice
- •For data annotation teams: Prioritize expert effort on texts where multiple AI annotators disagree to speed up codebook development for large labeled datasets.
- •For customer support analysts: Improve labeling guidelines for chat or transcript analysis by targeting expert review to cases with disagreements among automated annotators.
Authors
Zeyu He, Zhuqian Zhou, Kirk Vanacore, Rene F. Kizilcec, Ting-Hao 'Kenneth' Huang
Abstract
Large-scale text annotation brings expert insight to millions of documents, often through a codebook that AI annotators follow. Developing a robust codebook, however, takes months. Large language models (LLMs) could speed this process by applying an early codebook to the data, surfacing cases with strong LLM disagreement, and eliciting expert feedback to address them. We examined three ways experts can provide feedback for LLM codebook revision: (i) editing LLM-generated revisions driven by cross-LLM disagreement (Codebook Verifying), (ii) answering questions about LLM disagreements (Question Answering), and (iii) labeling disagreement cases with rationales (Rationale Labeling). Experiments on thousands of tutoring-session transcripts show that Rationale Labeling yielded the highest LLM-labeling accuracy (64.9%) against expert labels, outperforming the expert-revised codebook (57.8%). The best Question Answering setting also outperformed it (60.5%). Our work shows that LLMs can be used to strategically target expert attention, shortening months of codebook revision to days without sacrificing labeling performance.