Too Much of the Same: From Algorithmic to Human Bias in Learning to Defer
Human-Computer Interaction
Summary
The authors studied a way for AI models to pass harder decisions to humans, called Learning to Defer (LtD). They found that in tasks where some groups are smaller (imbalanced), the AI tends to pass more difficult cases from these smaller groups to humans. This can trigger human biases and make the humans' decisions worse, especially when the deferred cases are unevenly distributed. A user study showed that people made more mistakes when the cases they reviewed were very unbalanced, likely because their expectations didn’t match the actual case distribution. The authors highlight that these effects should be considered when using LtD in real-world settings.
Authors
Dario Pesenti, Alessandro Bogani, Stefano Teso, Andrea Pugnana
Abstract
Learning to Defer (LtD) extends supervised learning by allowing a Machine Learning (ML) model to defer harder or less confident decisions to a human expert. Despite being geared for human-AI collaboration, LtD strategies neglect the potential negative interference of human cognitive biases. Our contribution is twofold. First, we demonstrate that standard LtD strategies show class-dependent sampling bias in classification tasks in practice, and thus may disproportionately defer the minority classes when applied to imbalanced datasets. Second, we show that such asymmetries in task delegation may trigger human biases, ultimately leading to poorer downstream decision making. Specifically, we conduct a user study ($N=226$) where participants complete a classification task on a set of deferred items, with conditions presenting different levels of class imbalance. Our results show that participants exposed to a highly imbalanced rejection set achieved lower classification accuracy in the majority class compared to those exposed to a more balanced set, regardless of which class constituted the majority. Exploratory analyses suggest that this may be an instance of the Test-taker's effect, which stems from a mismatch between the actual distribution of classes and the participants' expectations about that distribution. Finally, we discuss the implications of these findings for the deployment of LtD algorithms.