TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks
2026-06-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors highlight that while machine learning models for tables have improved, the way features are prepared is often overlooked in evaluations. They create TabPrep, a simple tool that generates useful new features by focusing on three key patterns found in data. Their tests show that adding these features helps many types of models perform better, sometimes more than just improving the models themselves. TabPrep is also more efficient and works well across different datasets, making it easier for researchers to include feature engineering in their tests.
tabular machine learningfeature engineeringpreprocessing pipelinetree-based modelsneural networkslinear modelsbenchmarkingautomated feature engineeringdata patternsmodel evaluation
Authors
Andrej Tschalzev, Nick Erickson, Yuyang Wang, Huzefa Rangwala, Stefan Lüdtke, Heiner Stuckenschmidt, Christian Bartelt
Abstract
Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remains a critical yet underexplored component of real-world modeling pipelines that is entirely absent from modern benchmarks, which creates an unquantified evaluation gap. In this work, we introduce TabPrep, a lightweight preprocessing pipeline composed of feature generators that are carefully designed to target three specific structural data patterns. We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance. Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models, often surpassing gains achieved by model-centric innovations alone. TabPrep outperforms previous automated feature engineering approaches in performance, efficiency, and applicability across datasets, enabling integration into large-scale benchmarks. By releasing TabPrep (see https://github.com/atschalz/tabprep), we enable researchers to integrate feature engineering into their benchmarking setup, filling a longstanding gap in tabular evaluations.