Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to improve transfer learning with Tabular Foundation Models, which struggle when source and target data differ or when context size is limited. They propose a method called TL-ANDI that smartly picks important data points (anchors) from the source, based on how well they match the target data. These anchors are given refined labels and combined with a calibration step using the target data to make the transfer more reliable. This approach helps reduce problems caused by mixing different data sources without careful selection.
Tabular Foundation Modelstransfer learningin-context learningdistribution shiftoptimal transportposterior compatibilitydistillationcalibrationcontext size constraint
Authors
Yijun Lin, Sai Li
Abstract
Tabular Foundation Models (TFMs) have demonstrated strong empirical performance as black-box inference engines through in-context learning. However, their use in transfer learning is limited by two obstacles: strict context-size constraints and sensitivity to distribution shifts between source and target tasks. Directly pooling heterogeneous source data can therefore lead to negative transfer. To address these challenges, we propose Context-Constrained Transfer Learning via ANchoring and DIstillation (TL-ANDI), a posterior-aware distillation framework for TFMs. TL-ANDI constructs a compact source context by solving a budget-constrained optimal transport problem whose cost jointly measures target covariate coverage and posterior compatibility. The selected anchor samples are then equipped with locally distilled labels and combined with a residual calibration step using target data.