Additive regression methods improve prediction with unknown dependencies
Minimax Additive Regression under Unknown Dependent Designs
Machine Learning
Summary
Predicting outcomes when multiple factors influence a result can be tricky, especially if those factors are related to each other in unknown ways. The authors study a type of prediction model called additive regression that takes these relationships into account even when the data points are dependent rather than independent. They create new mathematical tools to measure and control these dependencies, allowing more accurate predictions even when the detailed data properties are unknown. Their work also shows it’s possible to recover the underlying contribution of each factor with good accuracy.
What this means in practice
- •For data scientists: Develop predictive models that remain accurate when input features are statistically dependent and marginal data distributions are unknown.
- •For financial analysts: Improve risk assessment models that rely on multiple, dependent economic indicators without full knowledge of their distributions.
Authors
Baptiste Ferrere, Fabrice Gamboa, Jean-Michel Loubes
Abstract
We study additive regression under a potentially non-product random design on $[0,1]^d$, allowing the dimension $d$ to grow with the sample size $n$. We introduce coupled smoothness classes that separately control the regularity of the marginal densities and the density-weighted additive components. To handle dependence, we adapt a Riesz-basis construction for functional ANOVA models and establish compatibility bounds with constants independent of the dimension under uniform bounds on the joint density. We construct thresholded least-squares estimators and establish matching minimax upper and lower bounds for prediction with known or unknown marginal densities, under suitable dimension-growth conditions. When the marginal densities are at least as smooth as the weighted components, the unknown-density problem attains the known-density minimax rate. When the densities are less smooth, their regularity determines the minimax rate over the coupled class. Finally, we show that the centered additive components can be recovered at the same aggregate upper rate, without an additional order of error.