New methods clarify compression limits for gaussian mixture data
Rate-Distortion Theory of Gaussian Mixtures via Posterior Label Sampling
Information Theory
Summary
This paper deals with understanding the best ways to compress data that comes from a mixture of different bell-shaped distributions in multiple dimensions. The authors propose a new method called Gibbs-relabelling that nearly achieves the best possible compression in many cases by clever sampling of hidden labels. They also suggest a complementary approach for very low rates of data compression. Their results help clarify when these methods reach fundamental limits on how well the data can be compressed without losing quality. They test their ideas using theoretical benchmarks and numerical examples.
What this means in practice
- •For data compression engineers: Design codecs that optimally compress data modeled as Gaussian mixtures using posterior label sampling and channel simulation techniques.
- •For signal processing developers: Improve rate-distortion performance in multicomponent signal compression by implementing stochastic quantization at low bit rates.
Authors
Weichen Wang, Deniz Gündüz
Abstract
We study the mean-squared error (MSE) rate-distortion function (RDF) of finite Gaussian mixture models (GMMs) in $\mathbb{R}^d$, for which neither the RDF nor an optimal test channel is known in closed form. It can be sandwiched between the genie-aided conditional RDF $R_{\mathbf{X} | T}(D)$, which can be characterized in closed-form for a GMM, and $R_{\mathbf{X} | T}(D)+I(\mathbf{X};T)$, where $T$ denotes the latent component label. We show that the upper bound can be achieved operationally by the proposed Gibbs-relabelling method (GRM), which conveys a label from the posterior $P_{T|\mathbf{X}}$ through \textit{channel simulation} at rate $I(\mathbf{X};T)$, followed by vector quantization of the resulting exactly Gaussian conditional sources by reverse water-filling. Using the Shannon lower bound (SLB), we prove that GRM attains the SLB, and hence the RDF, of a general finite mixture if and only if every component attains its own SLB. For a GMM this holds throughout the fully active interval $D/d<\min_{k,m}λ_{k,m}$, where $λ_{k,m}$ are the eigenvalues of the component covariances, and for scalar GMMs the condition is also necessary. Since GRM cannot operate below the label cost $I(\mathbf{X};T)$, we propose a complementary stochastic scalar quantization (SSQ) scheme for low rates, which simulates an $M$-ary reconstruction index. We show that at low rates SSQ with $M\geq 2$ matches the Shannon RDF to first order: $R(D_{\max}-δ)=δ/(2τ^2)+o(δ)$, where $τ^2$ is a directional sub-Gaussian proxy of the mixture, and quantify the price of a finite codebook through a strict rate gap at high rates and a distortion floor whose joint scaling in $M$ and number of mixture components is characterized. Numerical results illustrate both regimes and the finite-blocklength cost of channel simulation.