Improve sampling in complex high dimensional models with geometry aware monte carlo
Geometry-Aware Score-Repellent Monte Carlo
Machine Learning
Summary
Sampling methods help computers explore complicated mathematical landscapes, but can struggle when these landscapes have both steep and flat directions. The authors improve an existing method called Score-Repellent Monte Carlo by using geometric information to adapt how the sampling avoids past locations. This makes the method much better at navigating tricky shapes, especially when directions behave very differently. Their approach updates automatically during sampling and consistently performs better in tests on various real and simulated problems.
What this means in practice
- •For statistical modelers: Improve accuracy and efficiency of Bayesian inference in complex models with uneven parameter sensitivity by using geometry-aware score-repellent sampling.
- •For machine learning engineers: Enhance training of probabilistic models by better sampling from posterior distributions with challenging geometries, reducing error and runtime.
Authors
Jie Hu, Lingyun Chen, Do Young Eun
Abstract
The recently proposed Score-Repellent Monte Carlo (SRMC) method improves MCMC sampling by tilting the target density $π$ in $\mathbb{R}^d$ with the factor $\exp(-α\,θ_n^\top s(x))$, where the score function $s(x) = \nabla_x \log π(x)$, and $θ_n$ is the score average from past samples. However, a single scalar repellence strength $α\ge0$ cannot handle both the steep and the flat directions in anisotropic energy landscapes at once. We introduce \emph{Geometry-Aware SRMC} (GA-SRMC), which generalizes the Euclidean alignment to $θ^\top M s(x)$ for any symmetric positive-definite matrix $M$. A stochastic-approximation central limit theorem under a trace constraint on $M$ identifies the trace-normalized inverse Fisher $M_{\rm opt}\propto S^{-1}$ as the unique minimizer of the worst-direction covariance bound, elevating the Fisher information $S=\text{Cov}_π(s,s)$ from a proposal-side preconditioner (e.g., FisherMALA, natural gradient) to the bound-optimal repulsion-side matrix, with a single online estimate $\widehat S_n$ serving both roles at no additional computational cost. The full-Fisher online metric $M_n$ converges almost surely to $M_{\rm opt}$ and satisfies a CLT, providing a theoretical guarantee for the online implementation of the GA-SRMC framework. On an $8$-target simulation spanning the condition number $κ\in[1.3,100]$ and dimension $d\in[2,785]$, GA-SRMC dominates SRMC in ESS by $\boldsymbol{4.3\times}$ on anisotropic Gaussian ($κ=100$) and $\boldsymbol{6.8\times}$ on MNIST under the MALA baseline, by $\boldsymbol{16\text{--}32\%}$ on three well-conditioned logistic-regression posteriors, and by up to $\boldsymbol{+48\%}$ under FisherMALA; gains peak on the most anisotropic targets, and posterior-mean MSE is lower than with SRMC in all $16$ full-Fisher implementations.