Information bottleneck solutions for stable variable relationships
On the Information Bottleneck for Stable Variables
Information Theory
Summary
This paper studies a way to simplify information when two related data sources are connected with certain types of noise called stable noise. The authors found an exact way to encode this information using combinations of linear functions and stable noise, extending previous knowledge from the simpler Gaussian case. They also show that these encoding methods work best when compressing a lot of data but don't always reach the best possible performance otherwise. Their work generalizes well-known results and could help in understanding information compression beyond common assumptions.
What this means in practice
- •For signal processing engineers: Design encoding methods that optimally compress signals corrupted by stable noise beyond Gaussian assumptions.
- •For communication system designers: Improve theoretical models for encoding signals with stable noise characteristics to enhance data compression strategies.
A theory result. No direct application yet.
Authors
Dongmin Park, Jihad Fahs, Mohamad Assaad
Abstract
We consider a generalization of the Gaussian information bottleneck problem to jointly stable variables $X$ and $Y$. Specifically, for $X = Y + A$, where $A$ is stable and independent of $Y$, we find optimal stochastic linear encoders of the form $T = kX + N$, $N$ being stable and independent of $X$. We characterize the linear stable information bottleneck in closed-form and its critical value. Furthermore, we show that, except for the Gaussian case, such stochastic linear encoders are strictly suboptimal, however asymptotically optimal at large compression rate. Our linear solution recovers the Gaussian information bottleneck for jointly Gaussian variables.