De Bruijn Identity Research Articles

On the Equivalence Between Stein and De Bruijn Identities

This paper focuses on illustrating 1) the equivalence between Stein's identity and De Bruijn's identity, and 2) two extensions of De Bruijn's identity. First, it is shown that Stein's identity is equivalent to De Bruijn's identity under additive noise channels with specific conditions. Second, for arbitrary but fixed input and noise distributions under additive noise channels, the first derivative of the differential entropy is expressed by a function of the posterior mean, and the second derivative of the differential entropy is expressed in terms of a function of Fisher information. Several applications over a number of fields, such as signal processing and information theory, are presented to support the usefulness of the developed results in this paper.

An Alternative Proof for the Capacity Region of the Degraded Gaussian MIMO Broadcast Channel

We provide an alternative proof for the capacity region of the degraded Gaussian multiple-input multiple-output (MIMO) broadcast channel. Our proof does not use the channel enhancement technique as opposed to the original proof of Weingertan and the alternative proof of Liu et al. Our proof starts with the single-letter description of the capacity region of the degraded broadcast channel, and by using it, obtains a tight (i.e., achievable) outer bound for the capacity region of the degraded Gaussian MIMO broadcast channel, by using two main technical tools. The first one is the generalized de Bruijn identity due to Palomar which provides a connection between the differential entropy and the Fisher information matrix. The second tool we use is an inequality due to Dembo which lower bounds the differential entropy in terms of the Fisher information matrix.

Shannon-entropy-based nonequilibrium “entropic” temperature of a general distribution

The concept of temperature is one of the key ideas in describing the thermodynamical properties of systems. In classical statistical mechanics of ideal gases, the notion of temperature can be described in at least two different ways: the kinetic temperature (related to the average kinetic energy of the particles) and the thermodynamic temperature (related to the ratio between infinitesimal changes in entropy and energy). For the Boltzmann distribution, the two notions lead to the same result. However, for nonequilibrium phenomena, while the kinetic temperature has been commonly used both for theoretical and simulation purposes, there appears to be no corresponding general definition of thermodynamic or entropic temperature. In this paper, we consider the statistical or Shannon entropy of a system and use the "de Bruijn identity" from information theory (see Appendix A 2 for a derivation of this identity) to show that it is possible to define a "Shannon temperature" or "entropic temperature" T for a nonequilibrium system as the ratio between the average curvature of the Hamiltonian function associated with the system and the trace of the Fisher information matrix of the nonequilibrium probability distribution (see Appendix A 1 for a definition of the Fisher information). We show that this definition subsumes many other attempts at defining entropic temperatures for nonequilibrium systems and is not restricted to equilibrium or near equilibrium systems. Intuitively, the gist of our approach is to use the Shannon or Gibbs entropy of a system and make use of the relation dS=dQ(rev)/T as a definition of temperature. We achieve this by positing a statistical notion of infinitesimal heating as the addition of uncorrelated random variables (in a special way). As an example of the utility of such a definition, we obtain the nonequilibrium entropic temperature for a system satisfying the Langevin equations. For such a system, we show that while the kinetic temperature is related to the changes in the energy of the system, the entropic or Shannon temperature is related to the changes in the entropy of the system. We show that this notion, together with the well known Cramer-Rao inequality in statistics demonstrates the validity of the second law of thermodynamics for such a nonequilibrium system.

Information Theoretic Proofs of Entropy Power Inequalities

While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai and Verd\'u used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power.

On Mutual Information, Likelihood Ratios, and Estimation Error for the Additive Gaussian Channel

This paper considers the model of an arbitrary distributed signal x observed through an added independent white Gaussian noise w, y=x+w. New relations between the minimal mean square error of the non-causal estimator and the likelihood ratio between y and \omega are derived. This is followed by an extended version of a recently derived relation between the mutual information I(x;y) and the minimal mean square error. These results are applied to derive infinite dimensional versions of the Fisher information and the de Bruijn identity. The derivation of the results is based on the Malliavin calculus.

A Conditional Entropy Power Inequality for Dependent Variables

We provide a condition under which a version of Shannon's entropy power inequality will hold for dependent variables. We first provide a Fisher information inequality extending that found in the independent case. The key ingredients are a conditional expectation representation for the score function of a sum, and the de Bruijn identity which relates entropy and Fisher information.

Entropy and random vectors

Barron(1) produced a proof of the Central Limit Theorem for real-valued IID random variables, in the sense of convergence in relative entropy. Here, we establish a similar result for independent real-valued random vectors, not necessarily identically distributed. The main developments required are a generalisation of De Bruijn's identity, and various inequalities proposed in ref. 2.

A proof of the Fisher information inequality via a data processing argument

The Fisher information J(X) of a random variable X under a translation parameter appears in information theory in the classical proof of the entropy-power inequality (EPI). It enters the proof of the EPI via the De-Bruijn identity, where it measures the variation of the differential entropy under a Gaussian perturbation, and via the convolution inequality J(X+Y)/sup -1//spl ges/J(X)/sup -1/+J(Y)/sup -1/ (for independent X and Y), known as the Fisher information inequality (FII). The FII is proved in the literature directly, in a rather involved way. We give an alternative derivation of the FII, as a simple consequence of a "data processing inequality" for the Cramer-Rao lower bound on parameter estimation.