Method to measure shared and combined information in Gaussian data
Redundancy and synergy in multivariate Gaussians via the Blackwell order
Information Theory
Summary
Measuring how different sources share or combine information about something important is tricky, especially with complex data. The authors developed a new way to break down this information for Gaussian (bell curve) data using a concept called the Blackwell order, which tells us when one source is more informative than another. Their method gives a clear geometric picture, fast calculations, and matches some existing measures for two sources. They also show their approach works well even when looking at data with many dimensions or sources, and demonstrate it in a control problem involving sensors and memory.
What this means in practice
- •For machine learning engineers: Use the new method to identify overlapping and combined information from multiple sensors or features in Gaussian data for better model design.
- •For control system designers: Apply the technique to detect redundant and synergistic interactions in sensor and memory systems, improving optimal control strategies.
Authors
Artemy Kolchinsky
Abstract
The goal of the partial information decomposition (PID) is to quantify the redundant and synergistic information that multiple sources provide about a target. PID has many applications in machine learning, neuroscience, and other fields, but defining and computing it for high-dimensional continuous systems remains challenging. Here, we define a PID for multivariate Gaussian systems based on the Blackwell order, which formalizes when one channel is more informative than another. We prove that Gaussian channels are optimal for extracting both redundant and union information, yielding an intuitive geometric interpretation and an efficient numerical algorithm for the PID. Our union information and synergy coincide with the well-known BROJA measures, and we derive closed-form expressions for both in the case of two sources. We also argue that Blackwell redundancy (which differs from BROJA) is the only existing redundancy measure that satisfies a set of natural desiderata. We demonstrate the scalability of our method on systems with up to a thousand dimensions or sources. Our approach is illustrated on an optimal control problem, where it identifies redundant and synergistic interactions between sensor and memory.