Adding noise fixes bias in tree models for mixed data
Just add noise: Debiasing tree-based variable importance in mixed data
Machine Learning
Summary
Tree-based models like random forests often give too much importance to continuous data and overlook categorical data. The authors explain why this bias happens and suggest adding a tiny bit of random noise to categorical variables to fix it. They tested this idea on many datasets and combined it with a method to reliably select important variables while controlling errors. This makes the variable importance scores fairer and more useful when working with mixed types of data.
What this means in practice
- •For data science teams: Improve feature selection in predictive models that mix categorical and continuous data by reducing bias in variable importance scores.
- •For marketing analysts: More accurately identify key factors influencing customer behavior when datasets include categories like demographics along with numeric measures.
Authors
Jiahe Li, Omar Melikechi
Abstract
Variable importance scores from tree-based methods such as random forests favor continuous predictors over categorical ones. We present a theoretical analysis of this bias and propose a simple remedy: add a small amount of noise to each categorical predictor. The correction is demonstrated on a variety of simulated and real-world datasets and combined with integrated path stability selection to perform variable selection with false discovery control for mixed data.