Architecture choice improves ignoring irrelevant features in tabular ai models
Architecture Alignment With Sparse Priors in Tabular Foundation Models
Machine Learning
Summary
Some AI models analyze table data but get worse when extra useless columns appear. The authors studied different ways these models are built and found that models keeping separate awareness of each feature handle useless data better. They showed this by testing on both made-up and real data, and by comparing to theoretical ideal predictors. This means designing the model architecture to treat features individually helps it ignore distractions and focus on what matters.
What this means in practice
- •For data science teams: Improve predictive accuracy on datasets with many irrelevant features by choosing model architectures that preserve feature-specific information.
- •For business intelligence developers: Build tabular AI systems that robustly handle irrelevant columns without retraining, using architectures that align with sparse feature importance.
Authors
Tianqi Zhao, Tianyi Zhuang, Shuo Duan, Guanyang Wang, Yan Shuo Tan, Qiong Zhang
Abstract
Tabular foundation models (TFMs) are increasingly popular because they deliver strong predictions on new datasets through in-context learning, without task-specific training or extensive tuning. Yet released TFMs differ simultaneously in their pretraining priors, architectures, and objectives, obscuring their respective inductive biases. We therefore examine one concrete capability: irrelevant-feature suppression. Across synthetic tasks and real-world datasets, adding null features causes substantially greater predictive degradation in the row-token model TabDPT, whereas the cell-token alternating-axis model TabPFN v2 and other TFMs remain comparatively stable. This gap motivates us to ask whether architecture contributes to irrelevant-feature suppression. Because released TFMs remain confounded by other design choices, we train streamlined row-token and alternating-axis transformers under identical sparse-to-dense linear priors. Exact Bayes analysis shows that sparse prediction requires context-dependent feature gating, whereas the dense endpoint requires only uniform feature weighting. Consistent with this distinction, the alternating-axis model is substantially closer to the Bayesian optimal predictor on sparse tasks, while the architecture gap becomes negligible on dense tasks; almost all of the sparse gap arises from linear coefficient-estimation error. Finally, in both the controlled model and frozen TabPFN v2, we examine the effect of interventions on the feature-attention outputs on the linear coefficients, finding evidence of task-dependent selective routing of computation through feature-indexed pathways. Together, these results support architecture-prior alignment: preserving an addressable feature axis provides an inductive bias for task-adaptive relevance inference. Code is available at https://github.com/Tianqi-Zhao/ArchitecturePriorTFMs.