Abstract
We present novel data-processing inequalities relating the mutual information and the directed information in systems with feedback. The internal deterministic blocks within such systems are restricted only to be causal mappings, but are allowed to be non-linear and time varying, and randomized by their own external random input, can yield any stochastic mapping. These randomized blocks can for example represent source encoders, decoders, or even communication channels. Moreover, the involved signals can be arbitrarily distributed. Our first main result relates mutual and directed information and can be interpreted as a law of conservation of information flow. Our second main result is a pair of data-processing inequalities (one the conditional version of the other) between nested pairs of random sequences entirely within the closed loop. Our third main result introduces and characterizes the notion of in-the-loop (ITL) transmission rate for channel coding scenarios in which the messages are internal to the loop. Interestingly, in this case the conventional notions of transmission rate associated with the entropy of the messages and of channel capacity based on maximizing the mutual information between the messages and the output turn out to be inadequate. Instead, as we show, the ITL transmission rate is the unique notion of rate for which a channel code attains zero error probability if and only if such an ITL rate does not exceed the corresponding directed information rate from messages to decoded messages. We apply our data-processing inequalities to show that the supremum of achievable (in the usual channel coding sense) ITL transmission rates is upper bounded by the supremum of the directed information rate across the communication channel. Moreover, we present an example in which this upper bound is attained. Finally, we further illustrate the applicability of our results by discussing how they make possible the generalization of two fundamental inequalities known in networked control literature.
Highlights
We show that the supremum of the directed information rate across such a channel upper bounds the achievable ITL transmission rates
The directed information has been instrumental in characterizing the capacity of channels with feedback, as well as the rate-distortion function in setups involving feedback [5,20,21,22,29]
Theorems 7 and 8 imply that in the design of any encoder for in-the-loop messages, aiming to yield the joint probability distribution of channel input and output sequences that maximizes the directed information is of practical importance: it is necessary for achieving the highest “useful” transmission rate while minimizing the probability of error
Summary
The data-processing inequality states that if x, y, z are random variables such that x and z become independent when conditioning upon y, : iations. In this system, p, q, r, s, e, u, x, and y are random sequences, and the blocks S1 , . S4 represent possibly non-linear and timevarying causal discrete-time systems such that the total delay of the loop is at least one sample These blocks can model, for example, source encoders, decoders or even communication channels. We note that any of these exogenous signals, in combination with their corresponding deterministic mapping Si , can yield any desired stochastic causal mapping (for example, a noisy communication channel, a zero-delay source coder or decoder, or a causal dynamic system with disturbances and a random initial state)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
