Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data
Machine Learning
Summary
The authors address the challenge of transfer learning for tabular data, where feature types and meanings often differ between datasets, making it hard to share knowledge. They propose a new method called CATTLE that uses transformer model weights to learn general rules without needing the same features in both source and target data. Their experiments show CATTLE outperforms many existing methods across several datasets, improving prediction accuracy. The approach is data-agnostic and relies on interactions between transformer weights rather than typical data features.
Authors
Kazi F. Akhter, Ibna Kowsar, Manar D. Samad
Abstract
Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. \mds{This paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for $key$, $value$, and $query$ provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for $key$ from the source domain interact with the weight for $query$ in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods.} The CATTLE source code is available at https://tinyurl.com/pr5s8ywn.