Rotated trees improve regression by better fitting angled boundaries
RoBART: Bayesian Additive Regression Trees with Tree-Specific Rotations
Machine Learning
Summary
Standard methods that split data by looking straight up, down, left, or right can struggle to find patterns that aren’t lined up with these directions. The authors introduce RoBART, which lets each tree rotate its viewpoint before splitting, making it easier to fit data with tilted or curved patterns. They prove mathematically that this approach can learn certain smooth functions more quickly than regular methods. This means RoBART can better capture complex relationships when data isn’t aligned to simple axes.
What this means in practice
- •For data scientists: Improve regression models by capturing complex rotated boundaries that standard tree methods cannot fit well.
- •For financial modelers: Use rotated tree-based models to better predict outcomes when factors interact in non-axis-aligned ways.
Authors
Jeongung Heo, Seonghyun Jeong
Abstract
Bayesian additive regression trees (BART) can require many splits to approximate boundaries misaligned with the predictor axes. RoBART assigns each tree a rotation shared by all internal nodes, retaining axis-aligned splits in rotated coordinates and constant leaves. We jointly propose a Givens rotation sequence and cutpoints on the resulting grid by Metropolis-Hastings and establish reversibility with respect to the conditional posterior with leaf means integrated out. For additive functions with component-specific rotations and anisotropic Hölder smoothness, we prove posterior contraction in empirical $L_2$ distance and for the noise standard deviation. Under the stated prior, design, and grid conditions, with fixed numbers of predictors, trees, and components and no more components than trees, the rate is a sum of componentwise rates determined by smoothness and the number of rotated coordinates used. We also establish a posterior contraction lower bound showing that there exist functions for which RoBART adapts to the intrinsic dimension but axis-aligned BART does not.