Robustness measurement methods for naive Bayes and generative forest classifiers
Local Robustness Quantification for Naive Bayes Classifiers and Generative Forests: a General Approach
Machine Learning
Summary
This paper deals with measuring how stable the predictions of two types of machine learning models—naive Bayes classifiers and generative forests—are when their inputs slightly change. The researchers propose ways to quantify how much you can alter the model’s underlying assumptions before it changes its prediction. They tested these methods on standard datasets and found that the robustness measure can signal how much we should trust a given prediction. They also compared their approach to other ways of judging prediction reliability.
What this means in practice
- •For machine learning engineers: Evaluate and improve trust levels of predictions from naive Bayes and generative forest classifiers in real-world applications.
- •For risk assessment teams: Use robustness quantification to identify unreliable model predictions in critical decision-making contexts.
Authors
Adrián Detavernier, Jasper De Bock
Abstract
We provide methods for calculating the robustness of the predictions of two types of generative classifiers whose underlying distribution is a Probabilistic Graphical Model (PGM): naive Bayes classifiers and generative forests (a probabilistic extension of random forests). Following the paradigm of robustness quantification, we define the robustness of a prediction as the extent to which the distribution of the classifier can be perturbed without changing this prediction. We consider perturbations obtained by varying the local models of the PGMs within general neighborhoods and focus in particular on epsilon-contamination, total variation distance and chi-squared divergence balls. We test our methods on benchmark datasets, demonstrate that the robustness value of a prediction serves as an indicator for its trustworthiness and compare our approach with other such indicators.