Meta-learning trains small decision trees faster with less data
MotherTree: Meta-learning on synthetic data improves decision tree training
Machine Learning
Summary
Decision trees are simple models that are easy to understand and use for decision making but usually need to be built from scratch for every new problem. The authors introduce MotherTree, a method that learns how to build decision trees by training on lots of made-up data ahead of time. When given a small new dataset, MotherTree quickly creates a ready-to-use decision tree much like traditional ones, without starting over. It works well compared to standard tree-building methods, especially when data for a new task is limited, and can be further improved by fine-tuning.
What this means in practice
- •For machine learning engineers: Build interpretable decision tree models quickly for new tabular data tasks even when training data is limited.
- •For automated machine learning teams: Speed up decision tree training by initializing models with pre-trained meta-learned trees that improve small data performance.
Authors
Ziyuan Wang, Fredrik D. Johansson
Abstract
Conventional decision tree algorithms produce effective, transparent models that can be audited, communicated, and deployed independently of the training data, but require learning every new task from scratch. In contrast, tabular foundation models demonstrate that meta-learning from a synthetic prior distribution enables strong in-context prediction for previously unseen tasks, especially in small-sample regimes. However, this approach does not produce a standalone model that can be inspected in isolation. We introduce MotherTree, a tabular transformer that meta-learns decision tree induction: given a training set for a new task, it outputs a hard, axis-aligned decision tree, equivalent in form to classically trained trees, in a single forward pass. MotherTree is pre-trained on a synthetic prior using stochastic gradient descent without requiring reference trees for supervision. On established benchmarks with controlled sample size, the approach is competitive with size-matched trees from common algorithms: recursive partitioning, gradient-based tree learning, globally optimal trees, and distillation from tabular foundation models. Notably, MotherTree consistently improves over from-scratch gradient-based learning and acts as a strong initializer: task-specific tuning of the generated tree outperforms the corresponding from-scratch learner on all benchmarks and sample sizes. These results show that meta-learning can provide effective inductive biases for learning stand-alone, small decision tree classifiers.