Training models for clinical decisions improves some but not all predictions
Optimizing for the decision not the prediction: an exploration of Smooth Net Benefit as a training objective
Machine Learning
Summary
Prediction models in healthcare are usually trained to guess risks accurately, but decisions often depend on specific risk cutoffs. The authors created a new way to train models that focuses on these decision points instead of just prediction accuracy. They tested it with different models and found modest improvements mainly for simpler models, but no consistent benefits for more flexible ones. This suggests the new approach might help when models are limited in how well they can fit data.
What this means in practice
- •For clinical data scientists: Improve decision-focused risk predictions when using simpler models with limited flexibility, especially in clinical risk threshold settings.
- •For financial risk modelers: Adjust training objectives for logistic regression models to better capture threshold-based decision utility in financial risk assessments.
Authors
Koen M. F. Gorgels, Lasai Barreñada, Maarten van Smeden, Ben Van Calster, Ewout W. Steyerberg, Wouter A. C. van Amsterdam
Abstract
Objective Prediction models are commonly trained using objectives such as Bernoulli negative log-likelihood (NLL), although downstream clinical decisions may depend on specific risk thresholds. We introduce Smooth Net Benefit ($σ$NB), a differentiable approximation of Net Benefit designed to align model training with threshold-specific clinical utility. Materials and Methods We evaluated $σ$NB as a training objective for logistic regression, generalized additive models (GAMs), and XGBoost with three Hessian implementations. Experiments used the Framingham cardiovascular risk dataset and 44 TabZilla datasets comprising 72 dataset-threshold combinations. Results $σ$NB training did not consistently improve Net Benefit in Framingham. Across the TabZilla benchmark, mean standardized Net Benefit for logistic regression increased from 0.5669 with NLL to 0.5765 with $σ$NB (mean difference 0.0096, 95% CI -0.0001 to 0.0193). For GAMs, mean standardized Net Benefit decreased from 0.5921 to 0.5625 (mean difference -0.0296, 95% CI -0.0721 to 0.0129). For XGBoost, NLL achieved 0.6745 compared with 0.6723--0.6735 across $σ$NB implementations. In logistic regression, $σ$NB gains were positively associated with the performance advantage of XGBoost over NLL-trained logistic regression. Discussion The effect of $σ$NB was context dependent, with modest gains concentrated in logistic regression and little benefit for more flexible model classes. This suggests that decision-focused optimization may be most useful when limited model flexibility leaves greater scope for improvement. Conclusion Our results do not support $σ$NB as a general replacement for NLL training, but support further investigation of decision-focused objectives in settings where conventional likelihood-based training may not adequately capture decision-relevant structure.