Tabular foundation models perform well with fewer transformer blocks
Tabular Foundation Models Are Effectively Shallow
Machine LearningArtificial Intelligence
Summary
Tabular foundation models (TFMs), which use multiple transformer blocks to handle data arranged in tables, were thought to need many layers to perform well. This paper finds that most of these layers can be removed without much loss in prediction accuracy, making the models much smaller and faster. The authors show that even with half or almost all blocks replaced by a simple method, TFMs still beat traditional methods like decision trees on many tasks. Different TFMs rely on different parts of their architecture, but the main point is that these models are more complex than necessary.
What this means in practice
- •For data engineers: Reduce computational and memory costs of tabular transformers in production by pruning redundant blocks while maintaining accuracy.
- •For machine learning infrastructure teams: Speed up GPU inference of tabular models substantially by using simplified architectures with fewer transformer blocks.
Authors
Irene Cannistraci, Julia E. Vogt
Abstract
In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to scenarios with limited resources or privacy constraints, as they solve new tasks without additional training. However, their architecture consists of multiple transformer blocks, resulting in high computational and memory footprints. Meanwhile, traditional alternatives like Gradient-Boosted Decision Trees (GBDTs) remain highly competitive and run efficiently on CPUs, suggesting that tabular tasks may not inherently require deep architectures. This contrast raises a fundamental question: can TFMs be simplified by removing architectural redundancies without sacrificing performance? To answer this, we introduce a closed-form, label-free affine translator that replaces a contiguous window of up to 96% of the blocks, leaving the rest unchanged. Evaluating eight frontier TFMs on 171 classification datasets, we reveal that tabular transformers are vastly overparameterized: replacing half of a model's blocks costs at most $0.002$ median AUROC across all architectures, with the half-depth TabFM still outperforming every other full-depth TFMs and tuned GBDTs. Furthermore, different model families are overparameterized in different ways: TabPFN-3 relies on its encoders, maintaining performance regardless of which single block is retained. Other single-axis models spread computation thinly, needing only their first block, while dual-axis models concentrate it in one critical block. Ultimately, removing depth saves up to 1.5B parameters, speeds up GPU inference by up to $6\times$ at a median cost of $0.03$ AUROC, while a single-block TabPFN-3 serves cached queries $2.4\times$ faster using $6\times$ less memory at negligible cost.