Ranking method improves measurement of model calibration error
A Ranking Approach for Measuring Calibration
Machine Learning
Summary
Predictive models often estimate the chance of an outcome, but they aren't always perfectly accurate, a problem called miscalibration. The common way to measure this problem, called Expected Calibration Error, is hard to estimate accurately without making assumptions. The authors introduce a new way to measure calibration error, called rankECE, that compares predictions more locally and gives more reliable estimates. Tests show this new method works better than the usual approaches that group predictions into fixed bins.
What this means in practice
- •For machine learning engineers: Evaluate and improve the reliability of probabilistic predictions in classification models using a more accurate calibration error metric.
- •For data science teams: Assess predictive model outputs more precisely to enhance decision-making processes that depend on reliable probability estimates.
Authors
Anirban Chatterjee, Rina Foygel Barber
Abstract
When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that $Y=1$) exactly matches the forecasted probability $f(X)$. In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability $f(X)$. Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.