Bayes-optimal inference accuracy improves in large spherical linear models
TAP Accuracy Below the Fluctuation Scale and Universal Posterior Geometry in Spherical Linear Models
Machine Learning
Summary
This paper studies how to best estimate unknown values in large datasets modeled by spherical linear methods as more data points and features grow. The authors show mathematically that an existing approximation called the TAP approach closely matches the true best estimate with very small errors. They also analyze how the uncertainty or ‘shape’ of possible solutions looks and precisely how likely the estimates stray from a reference point. These results hold under realistic conditions on the data and help understand when and how Bayesian inference performs well in such models.
What this means in practice
- •For machine learning engineers: Optimize inference algorithms for models with spherical constraints using sharp TAP approximations to reduce errors in predictions.
- •For data scientists: Use precise bounds on uncertainty regions around estimates in high-dimensional linear regression under correlated designs to better interpret model certainty.
A theory result. No direct application yet.
Authors
Jingbo Liu, Zhiyuan Yu
Abstract
We study the Bayes-optimal spherical linear model as the ambient dimension and sample size grow proportionally, under a quantitative Marchenko--Pastur spectral-regularity condition on the design. This condition is satisfied by normalized i.i.d. designs with standardized entries of finite fourth moment, but does not require entrywise independence or impose conditions on the singular vectors. Under this condition, we prove a quantitative all-temperature TAP approximation and characterize the posterior geometry. For the natural finite-aspect-ratio TAP functional, the normalized spherical free energy and the TAP optimum differ by $O_P(p^{-1})$. Each is within $O_P(p^{-1/2})$ of its explicit deterministic equivalent, and this fluctuation scale is sharp. Uniformly over all global TAP maximizers, the normalized squared Euclidean distance to the spherical posterior mean is $O_P(p^{-1})$. We also prove that the posterior mass outside a data-dependent band determined by the ridge estimator has sharp exponential order. More precisely, uniformly over sufficiently small band widths $\varepsilon$, the logarithm of this mass is at most $-cp\varepsilon^2+O_P(1)$. For every fixed geometrically admissible width, a spherical-cap construction gives a matching exponential-order lower bound on this mass. For every deterministic sequence of widths $\varepsilon_p\gg p^{-1/2}$, the corresponding bands capture asymptotically all posterior mass.