Design-Based Supervised Learning with Noisy Human Labels
Artificial IntelligenceMachine Learning
Summary
The authors address the problem of errors in automated data labeling, where some human-reviewed labels are also noisy and only a portion are carefully checked by experts. They introduce a new method called PA-DSL that uses the expert-reviewed samples to fix noisy human labels and then improve the accuracy of statistical analysis based on all automated labels. Their method works well when the chance of audit and expert review is known, and tests show it reduces errors compared to only using expert-reviewed data. This helps better correct bias from imperfect labeling in data analysis.
Authors
Robert Chew, Matthew R. Williams
Abstract
Researchers increasingly use automated classifiers to label unstructured data for statistical analysis. Existing rectification methods can correct errors in these automated labels using a probability-sampled audit set, but they usually treat the audit labels as correct. In practice, human audit labels are often noisy, and only some audited items are reviewed by an expert or adjudicator. We propose Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method for this setting. It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels. The estimator is valid for a broad class of downstream analyses when the audit and adjudication probabilities are known. In synthetic and Wikipedia Detox semi-synthetic experiments, PA-DSL maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels when noisy human labels contain recoverable signal.