Model agnostic method selects important features from complex data
Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling
Machine Learning
Summary
It can be hard to figure out which features really matter in powerful but complex machine learning models. The authors introduce a method called LAMPS that works with any prediction model to find important features by testing how leaving out one feature at a time affects predictions. This approach cleverly focuses efforts on promising features while still exploring others and can pick out true important features even when data has lots of variables or correlated features. Experiments show it performs better than many current methods for choosing features.
What this means in practice
- •For data science teams: Select key predictive features from complex datasets without redesigning existing models.
- •For financial analysts: Identify important factors influencing predictions in models with many correlated market variables.
Authors
Xuhui Liu, Lili Zheng
Abstract
Black-box machine learning models increasingly deliver strong predictions, but extracting useful information from them, such as a set of important features, remains challenging. Existing model-agnostic methods primarily estimate feature importance or conduct inference on it rather than directly selecting features, whereas many feature selection methods are model-specific or rely on the model-X assumption. We introduce LOCO-guided Adaptive Minipatch Sampling (LAMPS), a model-agnostic ensemble framework that uses any black-box regression algorithm as its base learner to select features important for predicting the response. The base learner need only produce predictions and need not perform feature selection itself. LAMPS operates within a minipatch ensemble framework that subsamples both observations and features, allowing leave-one-covariate-out (LOCO) feature importance scores to be easily computed. It adaptively concentrates minipatch sampling on features with high LOCO scores while maintaining exploration. The resulting sampling probabilities rapidly separate signal from noise features after a few iterations, enabling selection through simple thresholding. We establish that LAMPS achieves exact feature selection in high-dimensional settings, provided that the base predictive models are sufficiently well trained on average. Extensive experiments on synthetic and real data show that LAMPS outperforms state-of-the-art feature selection methods, with particularly strong performance in the presence of correlated features.