Closed form solution speeds up gaussian information decomposition
Gaussian BROJA Partial Information Decomposition in Closed Form
Information Theory
Summary
Understanding how two sources share information about a target can be tricky, especially when dealing with many variables. The authors found a direct mathematical way to break down this shared information for Gaussian (bell curve) data without slow trial-and-error methods. This makes such analysis much faster and more practical for big data problems. They also made their code available for others to use.
What this means in practice
- •For machine learning engineers: Compute information decomposition of high-dimensional Gaussian features efficiently for feature selection and model interpretability.
- •For signal processing teams: Analyze redundancy and synergy between multiple sensor data streams to improve fusion algorithms using fast closed-form calculations.
Authors
Ohad Sheelo, Tal Golan, Ari Pakman
Abstract
The BROJA partial information decomposition provides an operationally motivated way to separate the information that two sources carry about a target into unique, redundant, and synergistic components. For continuous and high-dimensional variables, however, computing the BROJA decomposition requires solving a difficult optimization problem. Previous work showed that, when each source-target pair is jointly Gaussian, the problem can be reduced to finite-dimensional numerical optimization. Here, we derive a closed-form solution to this optimization problem, enabling direct computation of the BROJA decomposition in the general multivariate Gaussian setting. The resulting closed-form expression enables speedups exceeding two orders of magnitude for high-dimensional Gaussian BROJA PID. Code is available at https://github.com/brainsandmachines/Eigen_PID.