Human and language model team speeds text annotation guideline creation
CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation
Computation and LanguageArtificial Intelligence
Summary
Creating rules for labeling complex, specialized texts is often slow and difficult because experts need to agree on what to mark. The authors developed CoDA, a tool that lets experts work together with language models to quickly build these rule sets by reviewing mistakes and updating instructions. They tested CoDA on health notes and social media posts, finding it cuts guideline development from weeks to hours and uses fewer examples. This makes it easier to automatically find useful information in large sets of unstructured text.
What this means in practice
- •For health data teams: Rapidly build annotation guidelines to extract patient-related health insights from case notes using human-LM collaboration.
- •For social media monitoring teams: Develop targeted rules for sentiment detection in wildlife trafficking posts with minimal expert effort and fewer labeled samples.
Authors
Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante, Chandan Manjunath, Laura Onasch-Vera, Hailey Winetrobe Nadel, Eric Rice, Swabha Swayamdipta
Abstract
We present CoDA, an open-source human-LM collaborative codebook development assistant that helps domain experts develop annotation guidelines for annotating new variables in unstructured text with language models. Existing language model annotation systems primarily focus on downstream classification performance, while assuming the availability of high-quality annotation guidelines, or codebooks. In practice, developing these codebooks is time-consuming and challenging, particularly for concepts that require expert knowledge. CoDA addresses this gap by collaborating with domain experts to iteratively develop codebooks through targeted review of incorrect language model predictions and incorporating expert feedback into the evolving codebook. We demonstrate CoDA on two real-world case studies: extracting health indicators from homelessness case management notes and classifying sentiment of wildlife trafficking posts on social media. Across both domains, CoDA produces codebooks that achieve downstream classification performance comparable to expert codebooks while reducing codebook development time from several weeks to two hours and requiring 5X fewer manually annotated examples. Applying CoDA to generate annotations at scale, we identified 2,410 case notes that contained health discussions or referrals and 10,196 social media posts expressing negative sentiment that may warrant investigation for potential wildlife trafficking. By accelerating codebook development, CoDA helps practitioners use language models to extract domain-specific insights from unstructured text at scale.