Entropy and divergence concepts extended to non-invariant probabilities

Thermodynamic Formalism for Non-Invariant Probabilities: Entropy, Divergence, Channels

Information Theory

Summary

The paper deals with probability measures that do not follow traditional invariance rules in dynamical systems. The authors introduce new mathematical tools, like ni-entropy and ni-Kullback-Leibler divergence, to describe and measure these non-invariant probabilities. These tools extend existing concepts used in dynamics and information theory, allowing for a broader understanding and analysis of systems where probabilities evolve under more general channels. They also study how these new concepts connect to measures of pressure and information flow.

What this means in practice

  • For information theorists: Analyze communication channels with evolving, non-invariant probability distributions using the new entropy and divergence measures.
  • For statistical modelers: Model systems where probability measures shift dynamically without fixed points using the introduced thermodynamic formalism.

A theory result. No direct application yet.

Authors

Artur O. Lopes, Elismar R. Oliveira

Abstract

Let $\mathcal{M}$ and $\mathcal{M}_σ$ denote, respectively, the sets of Borel and $σ$-invariant probabilities on the uncountable set $Ω=\{1,2,...,d\}^\mathbb{N}$. Given a Hölder Jacobian $J$, the dual Ruelle operator $μ\in \mathcal{M} \mapsto \mathcal{L}_{J}^* (μ)$ will play the role of a dynamical channel; generalizes the action of a stochastic matrix on probability vectors. Since $\mathcal{L}_{J}^*$ generally breaks $σ$-invariance, we introduce the \emph{ni-entropy} $h_{ni}(μ)$ for non-invariant measures, employing the concept of the expected value function on cylinders of size one. $h_{ni}(μ)$ differs from Shannon entropy but extends the Kolmogorov-Sinai entropy from $\mathcal{M}_σ$ to $\mathcal{M}$; $h_{ni}$ is concave and u.s.c. This leads us to the concept of ni-pressure $\mathcal{P}(A)$ of a potential $A:Ω\to \mathbb{R}$. $μ\in \mathcal{M}$ is not a density with respect to a dominating measure. We define an {\it ni-Kullback-Leibler divergence} $D_{ni}(ρ|| ν)$ not assuming absolute continuity. We investigate concepts of Information Theory such as Minimal Output Entropy, Donsker-Varadhan variational principle, a variational formula for $h_{ni}$, bounds for divergence, information projections, and their connections to the ni-pressure $\mathcal{P}$.