Crowdsourcing label methods often give over or underconfident probabilities
On the Calibration of Crowdsourcing Aggregators
Machine Learning
Summary
When many people help label data, combining their answers can be tricky, especially when trying to guess how sure we should be. The authors show that simple ways to combine answers, like majority vote, often underestimate how confident they should be, while some advanced models are too sure of themselves. They prove that a Bayesian method is perfectly confident if the model matches the data, but others are usually off. They also suggest a new way to adjust scores after combining answers that makes confidence levels more accurate without hurting how often the answers are right.
What this means in practice
- •For data labeling teams: Improve confidence estimates in crowdsourced labels by applying post-hoc calibration to aggregated results, reducing under- or overconfidence bias.
- •For machine learning engineers: Build models with more reliably calibrated probabilistic labels from human or AI annotators by combining multiple aggregation methods and applying temperature scaling.
Authors
Gabriel Singer, Reda Arab, Maïssa Frikha, Samuel Gruffaz, Olivier Vo Van, Nicolas Vayatis, Argyris Kalogeratos
Abstract
Calibration has been extensively studied for classifiers, but remains largely unexplored for crowdsourcing aggregation methods, which are increasingly used for making probabilistic decisions. To the best of our knowledge, we provide the first theoretical and empirical study of calibration in label aggregation. To our great surprise, we find that widely used aggregation methods can be severely miscalibrated, and in opposite directions: Majority Vote is underconfident while model-based aggregators are overconfident. We prove that the Bayesian aggregator is perfectly calibrated under its data-generating model. We also show that exact calibration of weighted majority votes requires highly restrictive conditions, which helps explain why Majority Vote is generally miscalibrated in realistic crowds. Empirically, we study calibration across $39$ human crowdsourcing datasets and LLM-based crowds. In both cases, we observe systematic miscalibration for aggregation models such as Majority Vote, Dawid--Skene, GLAD, and CrowdFM. Finally, we propose a post-hoc calibration procedure that \emph{aggregates the aggregators}: for any aggregation function $φ$, we introduce a transformation that first combines its predictive distribution with those of complementary aggregators, before applying temperature scaling. This simple and interpretable transformation makes temperature scaling more effective than when applied directly to $φ$. It improves calibration, especially in the high-redundancy regime where there are many annotations per item, while generally preserving or even improving accuracy.