New dimension-free bounds clarify differences between statistical measures
Dimension-free comparison inequalities between $f$-divergences for Gaussian location mixtures
Information Theory
Summary
When comparing different ways to measure how much two sets of data differ, it can be tricky if the data lives in many dimensions. The authors found new mathematical relationships that hold no matter how many dimensions or components are involved, as long as the starting conditions are limited. These relationships help to understand and control differences between certain popular measures used in statistics and machine learning. Their approach uses ideas from how Gaussian shapes can change smoothly.
What this means in practice
- •For machine learning engineers: Analyze and bound errors in models based on Gaussian mixtures regardless of data dimension, improving reliability in uncertainty quantification.
- •For statistical data scientists: Use tighter bounds to compare different divergence measures in high-dimensional mixture models for model selection and hypothesis testing.
A theory result. No direct application yet.
Authors
Jeonghwan Lee, Cong Ma
Abstract
We establish dimension-free comparison inequalities between the Kullback-Leibler (KL) and $χ^2$-divergences and the squared Hellinger distance for Gaussian location mixtures with compactly supported mixing distributions. We show that both divergences are bounded by the squared Hellinger distance up to constants depending only on the radius of the support, independently of the ambient dimension and the number of mixture components. These results resolve an open question raised by \cite{jia2023entropic} and strengthen existing comparison inequalities. Our proof relies on a Gaussian interpolation argument that relates the derivatives of these divergences to differences between posterior means.