A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection
2026-07-27 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how crowdsourcing can be improved when certain labels (like rare classes) are harder to identify and annotators have different skills depending on the class. They created a new model that accounts both for how difficult an item is and how skilled annotators are for each class separately. Their work also examines how majority voting behaves when classes are imbalanced. Testing on many real-world datasets, their model better finds rare labels without losing overall accuracy, which is useful when catching rare but important cases is the goal.
crowdsourcingclass imbalanceannotator accuracyitem difficultygenerative modelminority recallmajority votingCondorcet's Jury Theoremmulticlass classificationlabel aggregation
Authors
Gabriel Singer, Samuel Gruffaz, Olivier Vo Van, Nicolas Vayatis, Argyris Kalogeratos
Abstract
We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones. In this setting, annotators may be reliable on both classes, unreliable on both classes, majority-class specialists, or minority-class specialists. Existing models only partially address this problem: they either capture class-dependent errors but ignore item difficulty, or they model item difficulty without capturing class-dependent errors. To fill this gap for imbalanced datasets in crowdsourcing, we introduce a generative aggregation model combining item difficulty with class-dependent annotator competence. The model allows both annotator abilities and item difficulties to vary across classes. We then revisit Condorcet's Jury Theorem in the class-imbalanced setting. We also show that majority voting asymptotically preserves the underlying class proportion. We evaluate our model on $33$ real-world crowdsourcing datasets, covering multiclass tasks such as images and text, as well as two large-scale regimes: large-scale annotation datasets, with many annotations per item, and large-scale item datasets, with a large number of annotated instances. Across these diverse settings, our model consistently achieves the highest minority recall while remaining competitive in balanced accuracy, making it particularly relevant when rare-label recovery is the primary objective.