Benchmark compares feature selection methods on real data
SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data
Machine LearningArtificial Intelligence
Summary
Choosing the right features from data tables is important in many fields, but current tests of the best methods often use outdated or fake data. The authors created SelectArena, a new benchmark using real-world datasets to fairly compare 15 common feature selection methods. They found certain methods are better at balancing accuracy, speed, and consistency, helping users pick the right approach for their needs. This benchmark can also be used to test new methods in the future.
What this means in practice
- •For data science teams: Use SelectArena results to choose feature selection methods that optimize model accuracy and training time for real-world datasets.
- •For climate modelers: Select interpretable feature subsets for understanding extreme weather factors using methods tested on realistic datasets.
Authors
Bastian Schäfer, Dominika Matus, Lennart Purucker, Ricardo Knauer, Erik Rodner, Frank Hutter, Anton Sokolov, Arnaud Liefooghe
Abstract
Selecting a subset of features (columns) from a tabular dataset is a ubiquitous challenge in real-world applications. Industry experts seek to minimize model training costs, climate scientists require an interpretable subset of factors to understand extreme weather events, and clinicians need to choose among diagnostic tests due to financial and time constraints. When looking at the scientific literature to determine which feature selection method to use, practitioners across domains face an unrepresentative landscape of benchmarking. Despite extensive research, existing feature selection benchmarks provide limited guidance because they rely on outdated models, ad hoc evaluation protocols, or artificially generated data that do not reflect real-world problems. To close the gap between scientific benchmarking and real-world applications, we introduce SelectArena, a modern benchmark for feature selection. We compare 15 widely used feature selection methods on 71 manually curated real-world tabular datasets, covering classification and regression problems, using standardized evaluation protocols and state-of-the-art tabular (foundation) models. Our results reveal clear trade-offs across predictive performance, runtime, stability, and validity. Tree-based methods, CART and RFImportance, along with F-Test are Pareto-optimal when taking into account predictive performance and runtime. F-Test, information-theory-, and distance-based methods lead on stability and validity, with F-Test emerging as a balanced choice across our metrics. Search-based methods are slow, unstable and achieve low performance. SelectArena turns these findings into practical guidance and, as an extensible platform, supports future feature selection research.