JABER improves quality control for human AI annotations with shared bias
JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias
Artificial Intelligence
Summary
Getting high-quality labels from humans to train AI is tricky because people can share the same biases, causing repeated judgments that seem right but might be wrong. The authors proposed JABER, a new system that carefully models both the true label and shared biases together to decide when to gather more labels or stop checking. They designed a plan to minimize mistakes and costs by considering the whole process, not just one step at a time. Their tests show JABER reduces errors and saves resources compared to older methods, even when biases make repeated checks unreliable.
What this means in practice
- •For machine learning engineers: Improve annotation pipelines by using JABER to reduce error rates and annotation costs when repeated judgments share biases.
- •For crowdsourcing platform managers: Deploy JABER to better decide when to request extra human or AI labels to ensure data quality without overspending.
Authors
Fengxu Yang, Minghui Liwang, Yang Lu, Bo Han, Xianbin Wang, Seyyedali Hosseinalipour
Abstract
High-quality annotations are essential for training and evaluating artificial intelligence (AI) systems. Quality control (QC) often acquires additional human or AI judgments before accepting a label. Nevertheless, shared biases can cause correlated judgments, leading conventional methods to mistake repeated agreement for independent evidence of correctness. To address this challenge, we propose JABER (Joint Alias-Preserving Belief for Evidence Routing), a bias-aware sequential QC framework that maintains a joint Bayesian posterior over label correctness and shared biases to guide evidence acquisition and termination decisions. Building on this posterior, we develop a finite-horizon algorithm that selects evidence-acquisition and termination actions to minimize expected QC loss under resource constraints. Our theoretical analysis shows that conditional-independence updating can become arbitrarily confident in an incorrect label under persistent shared bias. We further identify when repeated inspections cannot distinguish a correct label from an incorrect label supported by shared bias, when obtaining a new label can resolve this ambiguity, and why modeling the dependence between label correctness and shared bias can be necessary for QC decisions. Our experiments show that JABER reduces expected QC loss by an average of 4.65% compared with a method maintaining separate label correctness and bias probabilities when shared bias makes repeated inspections unreliable. Further, JABER's finite-horizon planning, which evaluates outcomes of successive QC actions, reduces expected QC loss by an average of 10.05% compared with a one-step lookahead method. In real-API experiments on FEVER, JABER achieves zero labeling errors while reducing QC costs by 2.67% and 16.99% compared with marginal-based belief modeling and one-step lookahead methods, respectively.