Classification with option to abstain under error limits
Classification with Abstention Under Class-Conditional Error Constraints
Machine Learning
Summary
This paper studies how to make decisions in situations where a system can choose not to decide rather than risk making mistakes. The authors focus on cases where mistakes affecting each type of decision are controlled separately, trying to reduce uncertainty while keeping errors low. They analyze how complex the decision rules can be and how many examples are needed to learn them reliably. To implement their ideas, they design computational methods compatible with models like neural networks and test these on real datasets, comparing with existing approaches.
What this means in practice
- •For machine learning engineers: Train classifiers that can abstain to meet strict error limits separately for each class, improving reliability in sensitive applications.
- •For medical device developers: Design diagnostic tools that decline to provide a diagnosis under low confidence while ensuring balanced error rates across conditions.$Commercial implications: Enables safer AI diagnostic products that meet regulatory safety thresholds by systematically controlling errors per condition.
Authors
Mohammadreza M. Kalan, Yuyang Deng, Sanaz Hamidi
Abstract
We study binary classification with abstention under separate class-conditional error constraints, with the objective of minimizing abstention while keeping both errors below prescribed thresholds. We characterize the distribution-free minimax rate of excess abstention risk, up to logarithmic factors, in terms of the complexity of the hypothesis class and the sample size. To make the framework amenable to computation with models such as neural networks, we introduce surrogate-loss formulations and derive finite-sample guarantees for excess surrogate ambiguity risk. We formulate the resulting learning task as a constrained optimization problem and characterize its computational complexity in the convex setting. Finally, we evaluate our approach on various datasets and compare its performance with a competing method for this problem.