Distilled models speed up and simplify tabular data predictions
Distillation of Tabular Foundation Models into Efficient Predictors
Machine Learning
Summary
Predicting outcomes from tables of data can be slow when using complex models that look at many examples at once. The authors show how to teach smaller, faster models to imitate these bigger models without losing much accuracy. They do this by training the small models on the big model’s predictions using both real and made-up data points. This method makes it cheaper and quicker to make predictions while keeping them reliable.
What this means in practice
- •For data science teams: Speed up prediction on tabular datasets by replacing slow foundation models with distilled lightweight models without losing much accuracy.
- •For business analysts: Use more efficient models distilled from complex foundation models for faster and cheaper data-driven decision-making.
Authors
Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo
Abstract
Tabular foundation models (TFMs) achieve strong predictive performance through in-context learning, yet repeatedly conditioning on labeled data makes inference expensive. Knowledge distillation can reduce this cost by transferring their predictive ability to lightweight, dataset-specific students. However, the dependence of TFM predictions on both a labeled context and a query introduces two design questions: how to construct teacher supervision and whether expanding query coverage improves distillation. We examine these questions across two TFMs and both neural and tree-based students, and derive an effective distillation recipe. The recipe uses the full labeled training set as teacher context and trains students solely on teacher predictions for observed and synthetic queries. On TabArena, the resulting students outperform their supervised trained tuned-and-ensembled counterparts by 57-98 Elo points. Applied unchanged to TALENT, the same recipe improves matched default students on 236-258 of 300 datasets and reduces median primary error by 4.0-6.4%. The distilled students also achieve median inference speedups of 3.0-21.6 times over their teachers, offering a practical trade-off between predictive performance and repeated inference cost. Code is available at https://github.com/nums-ai/TFM_Distillation .