information theorists

Abstract: Let $X$ be a random vector in $\mathbb{F}_2^m$, and let $X'$ be an independent copy. We prove that, for every $\varepsilon>0$, there is a subspace $G$ with $\dim G\le 2^{320(1+\varepsilon)/\varepsilon}H(X)$ such that $H(q_GX)\le(1+\varepsilon)(H(q_GX+q_GX')-H(q_GX))$. In particular, the quotient entropy is at most $(1+\varepsilon)(H(X+X')-H(X))$. The coefficient is independent of the ambient dimension, and its exponential order in $1/\varepsilon$ is necessary. The structural result is a logarithmic rigidity estimate for finite weighted families of laws with averaged subspace entropy sparsity. A common variational quotient controls both a family and its convolution family without decreasing the original family's normalized mean entropy defect. Posterior families preserve averaged sparsity, and a fresh-index argument combines a local-to-global estimate with the entropic Marton theorem to increase the defect ratio at a fixed multiplicative sparsity cost. We also obtain two-input almost-additivity and concentration of moderate-doubling sets on subspace cosets at the same exponential scale. The constants are not optimized.

Abstract: Partial information decomposition (PID) splits the information that several sources carry about a target into atoms, distinguished by which collections of sources give access to them. We ask how such a decomposition should behave when some of the sources are already known. The access structure of the atoms suggests one answer: discard the atoms that the known sources provide and regroup the rest according to which further sources unlock them. The classical construction of conditional information suggests another: recompute the decomposition under the conditional distributions and average. We show that the two answers agree, atom by atom, exactly when redundant information obeys a source chain rule, an analogue of the chain rule of mutual information: when the same sources are added to several collections, the enlarged collections share everything the added sources provide and, beyond that, exactly what the original collections share once the added sources are known. The rule divides established measures of redundant information, so that adopting or rejecting it genuinely narrows the choice of measure. Our main result is a no-go theorem: for three or more sources, the source chain rule and local positivity, the requirement that all atoms be nonnegative, cannot hold together in any PID.

Abstract: We show that the binomial distribution supported on $\{0,1,\ldots,n\}$ and with mean $np$ for some $0<p<1$ maximises entropy within the class of sums of negatively related indicator random variables with this mean and support. Our results generalise the main result of Yu (2008), who showed that this binomial distribution maximises entropy within the smaller class of ultra-log-concave random variables of order $n$. Our proof makes use of a Markov chain construction of Yu (2008), together with a stochastic ordering argument.

Abstract: We address an open problem raised by Polyanskiy and Wu in their textbook concerning the subadditivity of mutual \(f\)-information. We study two notions of generalized mutual information: mutual \(f\)-information, defined by \(\widetilde{I}_f(X;Y)=D_f(p_{XY}\|p_Xp_Y)\), and variational \((G,f)\)-information, obtained by minimizing an average of transformed \(f\)-divergences. For the first notion, we derive a sufficient condition on \(g(x)=x^2f''(x)\) that guarantees subadditivity under conditionally independent observations. This condition yields explicit families beyond the previously known examples generated by \(x\log x\) and \((x-1)\log x\). Our proof relies on the concavity of a generalized variance functional. For the variational notion, we extend existing sufficient conditions beyond the transforms \(G(x)\in\{x,\log(1+x),-\log(1-x)\}\). Using a comparison inequality for \(G^{-1}(G(A)+G(B))\), we obtain differential criteria for concave and convex transforms. Explicit constructions illustrate the resulting classes of subadditive information measures.

Abstract: We establish sharp weighted Rogers--Shephard inequalities for Rényi differential entropy. For independent copies $X,X'$ of a log-concave random vector in $\R^n$, we prove $h_p(aX-bX')\le h_p(X)+n\log(a+b)$ for all $a,b>0$ and $1\le p\le\infty$, where orders $1$ and $\infty$ correspond to Shannon entropy and min-entropy, respectively. The Shannon case settles the entropic Rogers--Shephard conjecture of Madiman and Kontoyiannis and its higher-dimensional formulation. For $1\le p<\infty$, equality holds precisely for invertible affine images of random vectors with independent exponential coordinates. At $p=\infty$, the larger equality class is characterized by possibly extended-valued convex potentials that are positively homogeneous of degree one. For every $0<p<1$, uniform laws on sufficiently high-dimensional simplices violate the unweighted bound. Thus $p=1$ is the critical order for this bound to hold in every dimension. The proof combines weighted convex intersections with an exponential lift to a log-affine probability density. At the Shannon endpoint, this lift yields an exact deficit decomposition with a conditional mutual information term for an additive channel with asymmetric Laplace noise.

Abstract: Wyner's common information $J(p)$ and the common entropy $G(p)$ of a pair of random variables $(X,Y)$ with finite ranges and joint distribution $p$ are both defined by an optimization over the auxiliary random variables $W$ that make $X$ and $Y$ conditionally independent: $J(p)$ is the infimum of the mutual information $I(X,Y;W)$, and $G(p)$ is the infimum of the entropy $H(W)$. We prove that both quantities are NP-hard to compute. For Wyner's common information, deciding whether $J(p)$ is strictly smaller than a given integer $r$ is NP-hard, and the problem is NP-complete on tables that are uniform on their support and have the same number of positive entries in every row. The reduction is from Maximum Edge Biclique and rests on an exact criterion, in terms of rectangles in the support of $p$, for improving on the auxiliary $W=X$; every yes-instance satisfies $J(p)<r-1/(4N^2)$ and every no-instance satisfies $J(p)=r$, where $N$ is the number of positive entries of $p$. For the common entropy, deciding whether $G(p)\le r$ for a rational number $r$ is NP-hard, even for tables with at most four nonzero entries in each row, all taking one of two values. The reduction is from exact cover by $3$-sets and rests on a majorization property of the masses of the nonnegative rank-one decompositions of $p$, combined with a quantitative form of the Schur concavity of entropy. In both cases the separation between yes- and no-instances is inverse-polynomial in the size of $p$, so that approximating $J(p)$ or $G(p)$ within a suitable inverse-polynomial additive error is also NP-hard. The two mechanisms are genuinely different: we exhibit a table on which a large rectangle in the support lowers $J$ below the threshold of our first reduction while leaving $G$ on it.

Abstract: We describe one explicit simple graph G on 32 vertices whose adaptive, normal, linear, entropic, and submodular widths are pairwise distinct. We compute all these widths exactly, except for the entropic width, where we only give a lower and upper bound. We use Ingleton's inequality and the Zhang-Yeung inequality for upper bounds, and give explicit constructions of modular, normal, linear, entropic, and non-entropic polymatroids for lower bounds.

Abstract: We derive explicit lower bounds for relative Fisher information by combining a variational principle with suitably orthogonalized Hermite-polynomial test functions. The resulting cumulant bounds are asymptotically sharp and yield lower bounds for Gaussian entropy deficits. We also establish quantitative versions of sharp min-entropy power inequalities in all dimensions. En route, we develop a stability result for Brzezinski's sharp bound for block sections of products of Euclidean balls, which may be of independent interest.