ReliableNet: A Chance-Constrained Approach to Trustworthy Classification in Deep Learning
2026-08-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address the problem of machine learning models being confidently wrong, which is a serious issue since such mistakes might not get flagged for human review. They introduce ReliableNet, a method that specifically limits the chances of making confident but incorrect predictions during training by using a special kind of optimization. Their tests across different kinds of data show ReliableNet reliably stays within its error limits and performs well even when data changes or is tricky. Compared to other methods, it better balances being accurate and knowing when it might be wrong. This work offers a structured way to build more trustworthy classifiers.
Empirical Risk MinimizationCalibrationSelective PredictionConfidenceJoint Confident-Wrong ProbabilityChance ConstraintsConformal Risk ControlRobustnessCoverageUncertainty Estimation
Authors
Ange-Clément Akazan, Ineza Remy Mugenga, Abebe Geletu, Jean Medard Ngnotchouye, Issa Karambal
Abstract
A prediction that is both confident and wrong is a critical reliability failure because it can bypass abstention and human review precisely when the model is mistaken. Empirical risk minimization (ERM) controls average loss but not this failure directly, while calibration, uncertainty estimation, conformal risk control, and selective prediction methods target related reliability properties rather than bounding the joint failure event during training. We propose ReliableNet, which constrains the Joint Confident-Wrong (JCW) probability, the probability that a prediction is simultaneously confident and incorrect, below a user-specified risk budget $α\in(0,1)$. We formulate this as a chance-constrained ERM problem, use a conservative smooth inner approximation whose population feasibility implies the original JCW constraint. Across four tabular and two image datasets, ReliableNet is the only method certified within the JCW budget for every dataset and seed in distribution, when compared against baselines spanning ERM, post-hoc calibration, conformal risk control, and selective prediction. Under demographic, ambiguity, spurious-correlation, novel-class, and covariate shifts, it achieves the lowest empirical JCW among the compared methods while remaining very competitive in accuracy, coverage, calibration, and selective prediction. Risk-coverage results further indicate that ReliableNet achieves better selective ranking than the benchmark methods on most datasets. Overall, ReliableNet provides a principled approach to trustworthy classification.