LLM annotators approach human consistency with new multi-agent method
Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge
Computation and Language
Summary
Annotating data with AI often struggles to match the variety and disagreement seen among human labelers, especially when guidelines are complex. The authors created a new benchmark with multiple human labels per example to better capture this disagreement. They also developed a multi-agent system that improves how AI models learn from contrasting feedback to better match human annotations. This method improves agreement accuracy and closely mimics human labeling styles across different tasks.
What this means in practice
- •For data annotation teams: Use the ANT-OR system to produce labeled datasets that better reflect human labeling variability and improve annotation consistency.
- •For natural language processing engineers: Incorporate multi-agent contrastive feedback methods to refine model annotations and adapt to complex, changing annotation guidelines.
Authors
Xiaopeng Ke, Zhixin Wang, Xuebo Liu, Xinyu Ma, Xing Zheng, Bowen Song, Weiqiang Wang, Min Zhang
Abstract
LLM-based annotation has advanced rapidly, yet existing benchmarks remain misaligned with real-world practice. Actual annotation often involves lengthy, task-specific guidelines, and different annotators may produce divergent but reasonable judgments, especially on boundary cases, where single-labeled dataset fail to reflect this disagreement. This gap also limits existing single-agent and workflow-based annotation systems, which often lack adaptability to complex, evolving guidelines. To bridge this gap, we introduce ANT-A Benchmark, a real-world multi-annotator benchmark spanning 7 datasets across domains and modalities, with task-specific guidelines and 5 independent human labels per instance. ANT-A enables a distribution-aware evaluation, using Fleiss kappa to assess how closely model annotations align with human labeling distributions. We further propose ANT-OR, a multi-agent annotation framework that refines prompts by distilling contrastive decision rules from positive and negative feedback. Experiments show that ANT-OR improves agreement@>=2 accuracy by 4.5 points over the strongest baseline and achieves the highest alignment with human annotator distributions as measured by Fleiss kappa within 2 points. Code and benchmark are available at https://github.com/Krueske/ANT-OR.