Tabular foundation models refine predictions using attention gated updates
What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates
Machine Learning
Summary
Many tabular datasets define new prediction tasks, which makes learning reusable models hard. The authors introduce a way for models to update their understanding during use by adjusting representations with known labels and applying these updates to predict unknown labels without changing the model itself. Their method uses attention mechanisms to focus updates and achieves better accuracy and efficiency than previous models on standard benchmarks. They show that these updates are essential for good predictions, revealing how task-specific knowledge builds up within the model during use.
What this means in practice
- •For data science teams: Use attention-gated updates to improve prediction accuracy on new tabular prediction tasks without retraining model parameters.
- •For business intelligence developers: Deploy refined in-context learning models to faster adapt tabular data predictions across diverse datasets in real time.
Authors
Tian Zhou, Beverly Jin, Linxiao Yang, Xue Wang, Wenwei Wang, Bingqing Peng, Mengni Ye, Jinjie Gu, Liang Sun
Abstract
What reusable computation should a tabular foundation model learn when every table defines a new supervised task? We develop in-situ representation refinement: support labels guide updates to the episode's representations, and these updates transfer to unlabeled queries without changing model parameters. A regularized leave-one-out objective yields a support correction and its query extension. The leading term separates attention-based reading from state-dependent scaling, motivating RefineICL: an attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed memory. RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation. It also improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8. Internal interventions show that support representations are more than a static source of labels: removing one intermediate support update, while preserving the query output, increases final query cross-entropy in all 72 tested episodes. Together, the derivation and interventions explain how attention-gated updates can construct a task-specific predictor in context.