Abstract
We provide a stochastic extension of the Baez–Fritz–Leinster characterization of the Shannon information loss associated with a measure-preserving function. This recovers the conditional entropy and a closely related information-theoretic measure that we call conditional information loss. Although not functorial, these information measures are semi-functorial, a concept we introduce that is definable in any Markov category. We also introduce the notion of an entropic Bayes’ rule for information measures, and we provide a characterization of conditional entropy in terms of this rule.
Highlights
IntroductionThe information loss K( f ) associated with a measure-preserving function (X, p) −→f (Y, q) between finite probability spaces is given by the Shannon entropy difference
The information loss K( f ) associated with a measure-preserving function (X, p) −→f (Y, q) between finite probability spaces is given by the Shannon entropy differenceK( f ) := H(p) − H(q), where H(p) := − ∑x∈X px log px is the Shannon entropy of p
The Bayesian inverse can be visualized using the bloom-shriek factorization because it itself has a bloom-shriek factorization f = πX ◦ f. This is obtained by finding the stochastic maps in the opposite direction of the arrows so that they reproduce the appropriate volumes of the water droplets. Given this perspective on Bayesian inversion, we prove that the conditional entropy f f of (X, p) (Y, q) equals the conditional information loss of its Bayesian inverse (Y, q)
Summary
The information loss K( f ) associated with a measure-preserving function (X, p) −→f (Y, q) between finite probability spaces is given by the Shannon entropy difference. K( f ) := H(p) − H(q), where H(p) := − ∑x∈X px log px is the Shannon entropy of p (and for q). In [1], Baez, Fritz, and Leinster proved that the information loss satisfies, and is uniquely characterized up to a non-negative multiplicative factor by, the following conditions: f. 0. Positivity: K( f ) ≥ 0 for all (X, p) −→ (Y, q). Positivity: K( f ) ≥ 0 for all (X, p) −→ (Y, q) This says that the information loss associated with a deterministic process is always non-negative
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
