Heavy-tailed data impact prediction error in high-dimensional learning
Asymptotic Analysis of Empirical Risk Minimization on Entry-wise i.i.d. Heavy-Tailed Data
Machine Learning
Summary
Some data in the real world have extreme values that happen more often than expected by normal math models. These big values affect how well computers can learn from data, especially when many features and samples are involved. The authors study a basic learning problem with such data and describe exactly how these extreme values affect prediction errors. They provide formulas that help understand and predict learning performance under this challenging situation. This work also shows general principles that apply to other problems with uneven or extreme local data variations.
What this means in practice
- •For machine learning engineers: Design linear models that account for unusually large feature values in high-dimensional data to improve prediction reliability.
- •For financial risk analysts: Improve modeling of heavy-tailed financial data to better estimate prediction errors in portfolio risk assessments.
Authors
Kaito Takanami, Takashi Takahashi, Yoshiyuki Kabashima
Abstract
Many real-world datasets exhibit unusually large values far more frequently than predicted by Gaussian models. Heavy-tailed distributions capture this behavior, yet evaluating learning performance under them remains challenging because rare, large feature entries retain non-vanishing effects even in high dimensions. Even in the canonical setting of empirical risk minimization for linear regression with entry-wise i.i.d. symmetric $α$-stable data, a precise asymptotic characterization of prediction has been lacking. In this work, we introduce a functional order parameter that describes the random effective problem associated with each coefficient. Using the replica method, we fully characterize the generalization error in the proportional high-dimensional limit where the sample size and feature dimension diverge at a fixed ratio. Additionally, this analysis establishes a heavy-tail universality law, scaling laws relating typical errors to prediction reliability, and the Bayes-optimal prediction error. In addition to characterizing the effects of extreme entries on the learning process, our method applies broadly to other systems with persistent local heterogeneity.