TabFM-Auto improves tabular data models with self-evolving pipelines
TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models
Machine Learning
Summary
Predicting from tables of data can be tricky because there are many ways to clean and organize the information. The authors created TabFM-Auto, which uses a language model to improve a base tabular model by automatically adjusting how data is prepared and processed using hints about the dataset. This approach adapts itself through trial and error, making predictions more accurate without retraining the model from scratch each time. Tested on many datasets and competitions, TabFM-Auto consistently improved performance and generated recipes that work well with other similar models too.
What this means in practice
- •For data science teams: Automatically optimize data cleaning and feature engineering steps for tabular prediction tasks using dataset metadata and feedback.
- •For business intelligence analysts: Deploy improved prediction pipelines that adapt to new structured datasets without retraining models from scratch.
Authors
Deqing Fu, Huangyuan Su, Rajat Sen, Taman Narayan, Sujay Sanghavi, Abhimanyu Das, Weihao Kong
Abstract
Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliary files that carry dataset semantics. Meanwhile, self-evolving machine learning engineering (MLE) agents train models from scratch on each dataset, yet jointly searching over features, architectures, and hyperparameters is noisy and prone to overfitting. We introduce TabFM-Auto, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it. Guided by dataset metadata and validation feedback, TabFM-Auto iteratively refines data cleaning, feature engineering, context selection, and post-processing to reduce TabFM's error. Across all 51 datasets of the TabArena benchmark, five TabFM-Auto configurations with different agents and language models take the top five overall positions, and the best raises TabFM from 1785 to 2013 Elo. The discovered pipelines also transfer to other frozen tabular foundation models (+69 to +143 Elo) with no further search. On the 8 tabular competitions of MLE-Bench, TabFM-Auto ranks first overall among MLE agents.