Shannon's Entropy Power Inequality Research Articles

The Entropy Power Inequality and Mrs. Gerber's Lemma for Groups of Order <inline-formula> <tex-math notation="TeX">${2^{n}}$ </tex-math></inline-formula>

Shannon's entropy power inequality can be viewed as characterizing the minimum differential entropy achievable by the sum of two independent random variables with fixed differential entropies. The entropy power inequality has played a key role in resolving a number of problems in information theory. It is therefore interesting to examine the existence of a similar inequality for discrete random variables. In this paper, we obtain an entropy power inequality for random variables taking values in a group of order 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> , i.e., for such a group G, we explicitly characterize the function f <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</sub> (x, y) giving the minimum entropy of the sum of two independent G-valued random variables with respective entropies x and y. Random variables achieving the extremum in this inequality are thus the analogs of Gaussians in this case, and these are also determined. It turns out that f <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</sub> (x, y) is convex in x for fixed y and, by symmetry, convex in y for fixed x. This is a generalization to groups of order 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> of the result known as Mrs. Gerber's Lemma.

On MMSE Crossing Properties and Implications in Parallel Vector Gaussian Channels

The scalar additive Gaussian noise channel has the “single crossing point” property between the minimum mean square error (MMSE) in the estimation of the input given the channel output, assuming a Gaussian input to the channel, and the MMSE assuming an arbitrary input. This paper extends the result to the parallel vector additive Gaussian channel in three phases. The channel matrix is the identity matrix, and we limit the Gaussian input to a vector of Gaussian i.i.d. elements. The “single crossing point” property is with respect to the signal-to-noise ratio (as in the scalar case). The channel matrix is arbitrary, and the Gaussian input is limited to an independent Gaussian input. A “single crossing point” property is derived for each diagonal element of the MMSE matrix. The Gaussian input is allowed to be an arbitrary Gaussian random vector. A “single crossing point” property is derived for each eigenvalue of the difference matrix between the two MMSE matrices. These three extensions are then translated to new information theoretic properties on the mutual information, using the I-MMSE relationship, a fundamental relationship between estimation theory and information theory revealed by Guo and coworkers. The results of the last phase are also translated to a new property of Fisher information. Finally, the applicability of all three extensions on information theoretic problems is demonstrated through a proof of a special case of Shannon's vector entropy power inequality, a converse proof of the capacity region of the parallel degraded broadcast channel (BC) under an input per-antenna power constraint and under an input covariance constraint, and a converse proof of the capacity region of the compound parallel degraded BC under an input covariance constraint.

On the Sum Rate of Gaussian Multiterminal Source Coding: New Proofs and Results

We show that the lower bound on the sum rate of the direct and indirect Gaussian multiterminal source coding problems can be derived in a unified manner by exploiting the semidefinite partial order of the distortion covariance matrices associated with the minimum mean squared error (MMSE) estimation and the so-called reduced optimal linear estimation, through which an intimate connection between the lower bound and the Berger-Tung upper bound is revealed. We give a new proof of the minimum sum rate of the indirect Gaussian multiterminal source coding problem (i.e., the Gaussian CEO problem). For the direct Gaussian multiterminal source coding problem, we derive a general lower bound on the sum rate and establish a set of sufficient conditions under which the lower bound coincides with the Berger-Tung upper bound. We show that the sufficient conditions are satisfied for a class of sources and distortion constraints; in particular, they hold for arbitrary positive definite source covariance matrices in the high-resolution regime. In contrast with the existing proofs, the new method does not rely on Shannon's entropy power inequality.

Monotonicity, Thinning, and Discrete Versions of the Entropy Power Inequality

We consider the entropy of sums of independent discrete random variables, in analogy with Shannon's Entropy Power Inequality, where equality holds for normals. In our case, infinite divisibility suggests that equality should hold for Poisson variables. We show that some natural analogues of the Entropy Power Inequality do not in fact hold, but propose an alternative formulation which does always hold. The key to many proofs of Shannon's Entropy Power Inequality is the behaviour of entropy on scaling of continuous random variables. We believe that R\'{e}nyi's operation of thinning discrete random variables plays a similar role to scaling, and give a sharp bound on how the entropy of ultra log-concave random variables behaves on thinning. In the spirit of the monotonicity results established by Artstein, Ball, Barthe and Naor, we prove a stronger version of concavity of entropy, which implies a strengthened form of our discrete Entropy Power Inequality.

Information Theoretic Proofs of Entropy Power Inequalities

While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai and Verd\'u used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power.

A simple proof of the entropy-power inequality

This correspondence gives a simple proof of Shannon's entropy-power inequality (EPI) using the relationship between mutual information and minimum mean-square error (MMSE) in Gaussian channels.

A Conditional Entropy Power Inequality for Dependent Variables

We provide a condition under which a version of Shannon's entropy power inequality will hold for dependent variables. We first provide a Fisher information inequality extending that found in the independent case. The key ingredients are a conditional expectation representation for the score function of a sum, and the de Bruijn identity which relates entropy and Fisher information.

Volumes of restricted Minkowski sums and the free analogue of the entropy power inequality

In noncommutative probability theory independence can be based on free products instead of tensor products. This yields a highly noncommutative theory: free probability theory (for an introduction see [9]). The analogue of entropy in the free context was introduced by the second named author in [8]. Here we show that Shannon's entropy power inequality ([6, 1]) has an analogue for the free entropy χ(X) (Theorem 2.1). The free entropy, consistent with Boltzmann's formulaS=klogW, was defined via volumes of matricial microstates. Proving the free entropy power inequality naturally becomes a geometric question. Restricting the Minkowski sum of two sets means to specify the set of pairs of points which will be added. The relevant inequality, which holds when the set of addable points is sufficiently large, differs from the Brunn-Minkowski inequality by having the exponent 1/n replaced by 2/n. Its proof uses the rearrangement inequality of Brascamp-Lieb-Lüttinger ([2]). Besides the free entropy power inequality, note that the inequality for restricted Minkowski sums may also underlie the classical Shannon entropy power inequality (see 3.2 below).

A new entropy power inequality

A strengthened version of Shannon's entropy power inequality for the case where one of the random vectors involved is Gaussian is proved. In particular it is shown that if independent Gaussian noise is added to an arbitrary, multivariate random variable, the entropy power of the resulting random variable is a concave function of the variance (power) of the added noise. The strengthened inequality is shown to hold for the class of stable distributions.