Bayesian method selects fair and accurate classifiers online
Bayesian Learning to Select Fair Classifiers
Machine Learning
Summary
Often, it's hard or expensive to get every expert or computer to label every example for tasks like sorting or recognizing things, especially when fairness matters. This paper studies how to pick the best labeler one by one as new tasks come in, balancing accuracy and fairness using a smart guessing method called Bayesian multi-armed bandits. The authors test their approach on real data with computers and human labelers and show it selects better labelers over time compared to simpler methods. They also provide math guarantees that their method works well considering both fairness and accuracy.
What this means in practice
- •For machine learning engineers: Select classifiers dynamically during deployment to ensure balanced accuracy and fairness on live data streams.
- •For data labeling teams: Adaptively choose human or AI labelers in cost-effective ways that maintain fairness in labeling sensitive data.
Authors
Saatvik Kher, Hyungrok Do, Judy Zhong, Padhraic Smyth
Abstract
Evaluations of classifiers (AI or human or agents) often assume a batch setting in which every agent produces labels on every example in a test set, which can be expensive or impractical in many contexts. This motivates online policies that adaptively select agents to query as examples arrive and update policies as outcomes are observed. We study the problem of selecting, in an online stream, the agent that best balances (i) predictive performance and (ii) subgroup fairness of a sensitive attribute, and that allows for delayed observations of true class labels. We formulate this as a Bayesian multi-armed bandit problem using Thompson Sampling, explicitly balancing accuracy and fairness. We conduct empirical evaluations on real-world datasets spanning problems involving tabular classifiers, large language models, and human labelers, demonstrating substantial reductions in regret relative to baselines. Our theoretical contributions include guarantees on the performance of the selected agent that incorporate both the accuracy and fairness aspects of the problem.