Feature folding reduces memory use and boosts tabular model accuracy
Support-Compiled Feature Folding: More Evidence at Lower Memory Across Tabular Foundation Models
Machine Learning
Summary
Tabular data models often struggle because analyzing all pairs of features uses too much memory, but selecting fewer features loses important information. The authors present a new method called Support-Compiled Feature Folding (SCFF), which rearranges how features are processed without changing the base model or adding training. This method reduces the memory needed while keeping or improving accuracy by focusing on the most important features in an efficient way. It works across multiple datasets and model types, making large tabular models more practical.
What this means in practice
- •For data engineering teams: Reduce memory usage and improve accuracy of tabular model inference on large datasets by applying SCFF without retraining models.
- •For cloud infrastructure operators: Lower peak GPU memory demands for tabular AI workloads, enabling more efficient resource allocation and cost savings.
Authors
Tian Zhou, Beverly Jin, Xue Wang, Linxiao Yang, Wenwei Wang, Bingqing Peng, Mengni Ye, Jinjie Gu, Liang Sun
Abstract
Tabular foundation models face a feature-side scaling dilemma: full-width pairwise mixing grows quadratically with the number of columns, whereas feature selection saves memory by discarding evidence. We introduce Support-Compiled Feature Folding (SCFF), a training-free inference framework that resolves this dilemma without changing the frozen backbone. SCFF routes support-ranked features through bounded leaves of the native feature encoder, support-checks the residual evidence, and merges the encoded messages before a single contextual prediction. It thereby converts quadratic feature-interaction work into linear-in-width work with a bounded local working set, without ensembling predictions or training new parameters. On the exhaustive 18-dataset wide-table slice of fixed AMLB-29, TabZilla, and TabArena snapshots, SCFF improves dataset-macro accuracy and NLL on all six evaluated backbones. All four matched-width comparisons retain favorable 95 percent dataset-bootstrap intervals on locked folds, with relative error reductions up to 26.1 percent. Median paired GPU-memory savings are 2.09x to 2.36x, and the ratio of separately observed maximum peaks reaches 34.3x. Under a measured peak-memory ceiling, SCFF uses the saved budget to preserve more support-selected evidence, improving accuracy by 4.06 and 3.72 points over the widest feasible single leaf on predeclared wide-Core strata of TabICLv2 and TabPFN-3.