Algorithms predict better for annotators who agree with others
Who Do Perspectivist Models Predict Well? Annotator Agreement, Sparsity, and Performance
Computation and Language
Summary
Some language tasks ask many people to rate the same text in their own way. The authors studied how well computer programs predict these human ratings, especially for people who often disagree with others or contribute fewer ratings. They found that algorithms guess better for people whose answers agree more with others. Surprisingly, how often a person rated did not make the algorithms more accurate.
What this means in practice
- •For nlp engineers: Improve natural language processing models by focusing training on annotators whose ratings align with others to increase prediction accuracy.
- •For content moderation teams: Adjust moderation feedback systems by recognizing that consistent annotators provide more reliable data for training decision algorithms.
Authors
London Lowmanstone, Vaagishwaran Sivakumar, Anwesha Samaddar, Akul Mundada, Harnoor Kaur, Maria Gini
Abstract
Successful completion of perspectivist natural language processing (NLP) annotation tasks requires algorithms to be aligned across diverse perspectives in order to predict how different humans will rate text. This work studies how the performance of such algorithms varies for annotators who behave differently than others, either by providing annotations that diverge from others' responses or by providing fewer annotations altogether. To aid with characterizing model performance, we curate the largest collection to date of perspectivist NLP datasets with ordinal annotations. After evaluating multiple classes of perspectivist algorithms across each of the datasets, we model the relationship between individual annotator idiosyncrasy and algorithm performance. In general, we find that algorithms perform better on annotators who tend to agree with others, but that annotator response rate does not have a statistically significant positive correlation with algorithm performance.