Data scaling effects vary across tabular machine learning models
On Data Scaling in Tabular ML: More Data, Better Generalization?
Machine Learning
Summary
This work looks at how adding lots more data affects the performance of machine learning models that work with tables of information. The authors study eight large real-world datasets to see how different models improve as they get more data. They find that decision tree and neural network models generally perform better with more data, but new foundation models behave less predictably. Also, when data differs between training and testing, performance can improve together at small scales but sometimes splits at large scales depending on the model and validation choices.
What this means in practice
- •For data science teams: Adjust training data size and model validation strategies when deploying tabular models under varying data conditions for better generalization.
- •For software engineers: Choose model types and training strategies informed by scaling behavior to optimize large-scale tabular ML applications.
Authors
Ivan Rubachev, Nikita Lazarev, Artem Babenko
Abstract
Recent work in tabular machine learning has emphasized the importance of diverse and realistic evaluation settings. The niche of large-scale (1M+) data, however, remains under-explored. To study training-data scaling, we assemble eight real-world tabular datasets with up to around ten million training examples, retaining the natural distribution shifts of the tasks and using random-split evaluation as a control. We find that GBDTs and MLP-based models show broadly similar scaling trends, with gains from both additional training data and algorithmic improvements. Tabular foundation models show more varied scaling behavior, with recent versions benefiting more consistently from larger contexts even under distribution shift. However, scaling training data under distribution shift introduces new challenges: in-distribution and out-of-distribution performance mostly improve together at smaller scales but can diverge at larger scales. We examine this through MLP capacity sweeps and comparisons of the validation distributions used for tuning and early stopping in MLPs and GBDTs. We also compare common practices of using validation data, contrasting its use for model selection in MLP pipelines with using it for in-context fitting in tabular foundation model pipelines. Under stronger shifts, MLPs selected using validation representative of the evaluation conditions generally maintain or improve performance as training data grows. In contrast, including such examples in context improves foundation-model performance, but does not prevent deterioration with increasing training size in our experiments.