Maximizing auc from biased positive unlabeled data using confidence scores
AUC Maximization from Biased Positive-unlabeled Data with Confidence
Machine LearningArtificial Intelligence
Summary
In many real-world problems, it is hard to get proper negative examples to train systems that distinguish between two types of things. This research looks at how to improve the accuracy of such systems when the available positive data is biased and the rest is unlabeled. The authors introduce a new way to use confidence scores—estimates of how likely examples are positive—to handle this bias and still learn effectively. Their method helps make better decisions even when the confidence scores are transformed versions of the true probabilities. They tested their approach on eight real datasets and showed it works well.
What this means in practice
- •For data scientists: Train binary classifiers effectively when high-quality negative samples are unavailable and positive data is biased, using confidence scores to improve performance.
- •For health data teams: Develop medical diagnostic tools where negative cases are rare or confidential, by using confidence-weighted positive samples to maximize classification performance.
Authors
Atsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi, Taishi Nishiyama, Kazuki Adachi, Yasuhiro Fujiwara
Abstract
Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced binary classification. Although positive and negative data are required for maximizing the AUC, negative data are often difficult to collect in some real-world applications due to privacy concerns or the need for specialized expertise to annotate them. Thus, AUC maximization from positive and unlabeled (PU) data has been attracting attention. Existing methods assume that labeled positive data are unbiased samples from the true positive distribution. However, this ideal assumption is often violated in practice. In this paper, we propose a method to maximize the AUC from biased PU data. To address the bias, our key idea is to exploit {\it confidence}, i.e., the probability that an instance is positive, associated with the small number of labeled positive data. We derive an estimator of the AUC risk using biased PU data with confidence, enabling AUC maximization under such bias. We further show that the rewritten AUC risk induces a Bayes-optimal AUC ranking even when the available confidence is any strictly increasing transformation of the true posterior probability. We experimentally show the effectiveness of our method on eight real-world datasets.