Tabular foundation models improve predictions by reusing past layers
Thinking in Depth: Retrospective Inference for Tabular Foundation Models
Machine Learning
Summary
Making predictions from tables can be tricky because the model needs to understand and compare many examples. The authors found that many models focus their improvements mostly in later processing steps, missing chances to improve earlier. They created a new model called Retro that looks back at earlier information while figuring things out, which helps it improve predictions more evenly through all steps. This approach lets different parts of the model handle different pieces of the problem, leading to better results on various tests.
What this means in practice
- •For data engineers: Design table prediction systems that reuse intermediate model states for better accuracy and efficiency on diverse tabular tasks.
- •For business intelligence teams: Improve automated insights from labeled tabular data by using models that integrate information from multiple processing stages.
Authors
Hao-Run Cai, Si-Yang Liu, Zi-Jian Cheng, Kun-Yang Yu, Jin-Hao Sheng, Guo Yu, Chonghan Liu, Zhi Zhou, Jun-Peng Jiang, Lan-Zhe Guo, Han-Jia Ye
Abstract
Tabular foundation models (TFMs) are pretrained across diverse tabular tasks and make predictions on a new table at inference time using its labeled examples as context. Most recent TFMs perform such in-context prediction with stacked Transformer layers, repeatedly transforming how examples are represented and compared. By tracing individual queries through several strong TFMs, we find that predictive refinement is highly uneven across depth and is often concentrated in later layers. This uneven refinement motivates us to reconsider how intermediate representations are constructed and reused throughout the network. We introduce Retro, a tabular foundation model based on retrospective inference, where later stages can explicitly revisit and recombine intermediate information produced earlier in the network. Retro organizes this process around two complementary operations: which intermediate information to revisit, and how the resulting contextual update should be shaped for each query. Attention Residuals address the former by adaptively reweighting contributions from different depths, while query-conditioned Gated Attention addresses the latter by modulating the attention output element-wise across representation dimensions. Our analysis shows that Retro shifts predictive refinement earlier and more broadly across depth, with different stages revising different subsets of queries in a pattern suggestive of multi-view refinement. Across TabArena, TALENT, and RelArena, Retro ranks among the top three and lies on the Pareto frontier. These results indicate that directly reusing intermediate representations provides a practical way to better exploit depth in TFMs.