Choosing a parallel heterogeneous ensemble method for tabular classification
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied different ways to combine multiple classifiers (ensemble methods) on 56 smaller datasets and created a set of recommendations for their use. They tested these recommendations on 28 new tasks and found that their approach worked better than simply choosing the best single model, and was as good or better than individual ensemble methods. They noticed that Blending and Stacking, two ensemble techniques, behave inconsistently but in different ways depending on the dataset. They also found that Hard Voting, while simple, gives less reliable probabilities than a method they call Robust Soft Voting, which works well especially when there are multiple classes to predict.
ensemble methodstabular classificationBlendingStackingHard VotingSoft Votingprobabilistic classificationmulticlass classificationOpenML CC18TabArena
Authors
Vassili Maillet, Gustavo, Angulo, Pierre Jouvelot
Abstract
Parallel ensemble methods were compared on $56$ small-to-medium tabular classification tasks drawn from OpenML CC18. A set of ``best practice'' recommendations on the use of ensemble methods was derived from these observations. It was later validated on 28 additional tasks using TabArena's precomputed data, where the recommendation set significantly outperformed Single Best and matched or exceeded individual ensemble methods. Two key observations were made. First, Blending and Stacking are inconsistent, but their inconsistencies are independent and happen on different tasks. Second, while Hard Voting's probabilistic classification is rather weak, a consequence of using vote proportions as posterior estimates, Robust Soft Voting's probabilistic classification is particularly successful, especially in the multiclass case.