Information links estimation and prediction for discrete channels
Information, Estimation, and Prediction for Discrete Memoryless Channels via Poissonization
Information Theory
Summary
This paper studies how information and estimation relate when data passes through a channel that transforms input symbols into output symbols. The authors imagine that the input is sent a random number of times following a Poisson distribution, which controls how much data is observed. They find that the change in mutual information corresponds exactly to the best possible accuracy in guessing the input from the output. Their work also characterizes how mistakes in assuming the input or channel affect prediction accuracy. Finally, they extend these ideas to sequences of inputs observed at random times.
What this means in practice
- •For communication system engineers: Design channel coding and estimation schemes by linking mutual information changes to decoding accuracy under variable observation budgets.
- •For signal processing teams: Improve prediction algorithms for discrete signals using the derived relations between observation count and estimation error under model mismatch.
A theory result. No direct application yet.
Authors
Abhiram Gorle, Tsachy Weissman
Abstract
We develop information--estimation relations for an arbitrary discrete memoryless channel with finite input and output alphabets by varying the expected number of observations while keeping the channel fixed. A single hidden input is observed through a Poisson number of conditionally independent channel uses, whose mean specifies the observation budget. The derivative of mutual information with respect to this budget equals the minimum expected relative-entropy loss for estimating the input's output distribution. The posterior average of the channel's output distributions achieves this minimum and is the Bayesian predictor for the next observation. When the input distribution, the channel, or both are misspecified, the relative entropy between the true and assumed observation laws is the integral of the excess prediction loss over the budget. We also extend these relations to finite-state processes taking values in the input alphabet and observed through the channel at independent Poisson sampling times. Both in the matched setting and under model mismatch, the expected prediction loss accumulated over time using past observations equals the corresponding loss using the full observation record (noncausal estimation), averaged over lower sampling rates.