Subadditivity conditions for generalized mutual information measures
On the Subadditivity of Mutual f-Information
Information Theory
Summary
This paper tackles a question about how certain ways of measuring information behave when combining independent sources. The authors study generalized versions of mutual information, which is a way to quantify how much two things are related. They find new conditions under which these information measures add up nicely, extending what was known before. Their results give clear rules for constructing new measures that behave predictably with independent data.
What this means in practice
- •For information theorists: Use the new subadditivity conditions to design or analyze information measures that combine data sources predictably under independence assumptions.
- •For machine learning engineers: Incorporate the expanded classes of generalized mutual information into algorithms that rely on measuring dependencies between variables for better performance guarantees.
A theory result. No direct application yet.
Authors
Hamidreza Abin, Amin Gohari
Abstract
We address an open problem raised by Polyanskiy and Wu in their textbook concerning the subadditivity of mutual \(f\)-information. We study two notions of generalized mutual information: mutual \(f\)-information, defined by \(\widetilde{I}_f(X;Y)=D_f(p_{XY}\|p_Xp_Y)\), and variational \((G,f)\)-information, obtained by minimizing an average of transformed \(f\)-divergences. For the first notion, we derive a sufficient condition on \(g(x)=x^2f''(x)\) that guarantees subadditivity under conditionally independent observations. This condition yields explicit families beyond the previously known examples generated by \(x\log x\) and \((x-1)\log x\). Our proof relies on the concavity of a generalized variance functional. For the variational notion, we extend existing sufficient conditions beyond the transforms \(G(x)\in\{x,\log(1+x),-\log(1-x)\}\). Using a comparison inequality for \(G^{-1}(G(A)+G(B))\), we obtain differential criteria for concave and convex transforms. Explicit constructions illustrate the resulting classes of subadditive information measures.