Binary classification prior improves multi-class tabular predictions
Two Is Enough: Binary Classification Priors Suffice for Multi-Class In-Context Tabular Prediction
Machine Learning
Summary
This paper finds that training a transformer model only on binary classification tasks can still allow it to handle multiple classes when making predictions on tabular data. The authors show that their simpler training setup performs better than previous methods on many datasets, including real-world credit scoring. This works because the model learns to compare data entries in context rather than relying on a fixed number of output classes during training. As a result, limiting training to two-class problems makes the model smaller and more flexible without losing accuracy.
What this means in practice
- •For credit risk analysts: Improve credit scoring models by using a pretrained tabular model trained on binary tasks to enhance classification accuracy across many institutions.$Commercial implications: Enables better credit risk products by improving predictive accuracy using efficient pretrained models suitable for financial institutions.
- •For data science teams: Simplify model pretraining by focusing on binary classification priors, reducing computational costs while maintaining strong multi-class prediction performance.
Authors
Zhixiong Cheng, Shiyuan Han, Jingjing Suo, Zhengli Wang
Abstract
Tabular foundation models (TFMs) pretrain a transformer on large-scale synthetic priors so that a single forward pass yields zero-shot tabular prediction. TabICLv2 samples the class count of its classification prior during pretraining, and the prevailing configuration caps it at ten. We report a counter-intuitive result: setting this cap directly to two---making every pretrained classification task binary---still yields weights that perform zero-shot K-way classification 2. On the 38 TabArena classification datasets (200 bootstrap rounds, anchored at default RF = 1000), the binary two-stage recipe reaches 1590.1 Elo, above the official default at 1568.0 (+22.1), above TabPFN-v2.6 (1577.1) and RealTabPFN-v2.5 (1521.6). A production-scale validation on 60 real-world credit-scoring institutions independently confirms the direction: the same checkpoint improves test AUC on 44 of 60 institutions (paired Wilcoxon p = 0.0007 for AUC, p = 0.0002 for KS) at +1.5 inference cost. Mechanistically, classification in TabICLv2 is carried not by a fixed-cardinality output head but by an in-context comparison operator matching query-row embeddings against class-conditional embeddings, so the cardinality of the output space is set by the labels present in the context, not by the label space seen during pretraining. This makes a binary prior structurally cheaper to generate and cache than a ten-class prior, demotes "number of classes" from an architectural constraint to a tunable hyperparameter, and points to a class-count-agnostic regime of TFM pretraining.