Risk models find patient groups with poor prediction accuracy
Local Discrimination of Binary Classifiers via Localized AUC Functional
Machine Learning
Summary
Risk prediction models are often evaluated using AUC, a score that shows overall accuracy but hides how well the model works for different groups of patients. The authors developed a new method called LOCATOR that uses a specialized score, Localized AUC, to find patient groups where the model performs poorly. They tested this method on both real patient data and simulated data to show it can detect these groups. This helps make sure decisions based on risk models are more reliable for everyone.
What this means in practice
- •For hospital data teams: Identify patient groups where clinical risk models may give inaccurate predictions to improve patient care decisions.
- •For healthcare software developers: Embed local accuracy measures like Localized AUC into risk prediction tools to flag cases needing review or alternative evaluation.
Authors
Wenye Ren, Anna Győrffy-Kerekes, Tancredi Dorsi, Olga Demler
Abstract
Risk prediction models are frequently evaluated with AUC, which means that during evaluation we are blind to performance heterogeneity, which could lead to bad decision-making in clinics. In this work, we aim to change that by identifying patient cohorts for whom the risk prediction model might not be an accurate predictor. To separate these cohorts, we use a version of the weighted AUC, which we call Localized AUC or LAUC. Using this metric, we build a method that we coin LOCATOR. This method is able to identify cohorts of patients for whom the risk prediction model is inaccurate. We test this method both in real-world data and in simulations. We also build a statistical background for the use of Localized AUC for our purposes.