CALICO makes annotation prompts editable and optimizable for experts
CALICO: A Human-Centered, Codebook-Aligned System for Annotation
Human-Computer InteractionComputation and Language
Summary
It is often hard for domain experts without technical skills to fix errors in AI that label data based on instructions called codebooks. The researchers created CALICO, a system that lets experts edit, version, and improve these instructions interactively to better match their rules. CALICO combines automatic tools and human feedback to adjust prompts used by AI models, improving accuracy and capturing individual annotators’ styles. The system was tested on chatbot conversations and showed significant improvements in labeling quality. It is available as an easy-to-use web application with code and demos publicly released.
What this means in practice
- •For data annotation teams: Edit and optimize AI labeling prompts in real time to improve annotation consistency and reflect expert guidelines accurately.
- •For customer support teams: Customize AI chatbots’ understanding by refining conversation labeling instructions aligned with specific business rules using an easy web interface.
Authors
Boqin Yuan, Xiaoyi Gu, Fiona Li, Chang Wan, Angel Hsing-Chi Hwang, Jieyu Zhao
Abstract
Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support for translating domain experts' codebooks into reliable, revisable, and auditable prompts. Prompts are often treated as fixed instructions and hidden from annotators, making it difficult for non-technical domain experts to diagnose and correct model behavior when outputs violate codebook guidelines. In this paper, we present CALICO, a human-centered, codebook-aligned annotation workflow that treats prompts as editable, versioned, and optimizable artifacts. CALICO integrates codebook parsing, prompt generation, result inspection, prompt versioning, natural language human feedback, and label-supervised prompt optimization through existing optimizers such as GEPA, MIPROv2, and OPRO, together with our reflection-based optimizer, ReflectAgent. Empirically, we evaluate CALICO on domain-specific AI-companion chatbot conversation codebooks. Across evaluated dimensions, CALICO improves mean held-out performance by +13.0 and +7.4 absolute points for two coders, respectively. A coder-specificity analysis further suggests that optimized prompts capture coder-specific interpretations rather than only generic codebook clarification. CALICO runs as a web application that takes users from raw codebook materials to inspectable, exportable labels; the website, codebase, and live demo are released at https://calico-annotation.github.io/ under the Apache 2.0 License.