Entropy Power Inequality Research Articles

Distributed Rate-Distortion With Common Components

We describe a scheme for rate-distortion with distributed encoding in which the sources to be compressed contain a common component. We show that this scheme is optimal in some situations and that it strictly improves upon existing schemes, which do not make full use of common components. This establishes that independent quantization followed by independent binning is not optimal for the two-encoder problem with a distortion constraint on one source. We also show that independent quantization and binning is suboptimal for the three-encoder problem in which the goal is to reproduce one of the sources losslessly. This provides a counterexample that is fundamentally different from one provided earlier by Korner and Marton. The proofs rely on the binary analogue of the entropy power inequality and the existence of a rate loss for the binary symmetric Wyner-Ziv problem.

Randomized Resource Allocation in Decentralized Wireless Networks

In this paper, we consider a decentralized wireless communication network with a fixed number u of frequency subbands to be shared among N transmitter-receiver pairs. It is assumed that the number of active users is a realization of a random variable with a given probability mass function. Moreover, users are unaware of each other's codebooks and hence, no multiuser detection is possible. We propose a randomized frequency hopping (FH) scheme in which each transmitter randomly hops over a subset of u subbands from transmission slot to transmission slot. Assuming all users transmit Gaussian signals, the distribution of the noise plus interference is mixed Gaussian, which makes calculation of the mutual information between the transmitted and received signals of each user intractable. We derive lower and upper bounds on the mutual information of each user and demonstrate that, for large signal-to-noise ratio (SNR) values, the two bounds coincide. This observation enables us to compute the sum multiplexing gain of the system and obtain the optimum hopping strategy for maximizing this quantity. We compare the performance of the FH system to that of the frequency division (FD) system in terms of the following performance measures: average sum multiplexing gain (η(1)) and average minimum multiplexing gain per user (η(2)). We show that (depending on the probability mass function of the number of active users) the FH system can offer a significant improvement in terms of η(1) and η(2) (implying a more efficient usage of the spectrum). In the sequel, we consider a scenario where the transmitters are unaware of the number of active users in the network as well as the channel gains. Developing a new upper bound on the differential entropy of a mixed Gaussian random vector and using entropy power inequality, we obtain lower bounds on the maximum transmission rate per user to ensure a specified outage probability at a given SNR level. We demonstrate that the so-called outage capacity can be considerably higher in the FH scheme than in the FD scenario for reasonable distributions on the number of active users. This guarantees a higher spectral efficiency in FH compared to FD.

Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error

Consider the minimum mean-square error (MMSE) of estimating an arbitrary random variable from its observation contaminated by Gaussian noise. The MMSE can be regarded as a function of the signal-to-noise ratio (SNR) as well as a functional of the input distribution (of the random variable to be estimated). It is shown that the MMSE is concave in the input distribution at any given SNR. For a given input distribution, the MMSE is found to be infinitely differentiable at all positive SNR, and in fact a real analytic function in SNR under mild conditions. The key to these regularity results is that the posterior distribution conditioned on the observation through Gaussian channels always decays at least as quickly as some Gaussian density. Furthermore, simple expressions for the first three derivatives of the MMSE with respect to the SNR are obtained. It is also shown that, as functions of the SNR, the curves for the MMSE of a Gaussian input and that of a non-Gaussian input cross at most once over all SNRs. These properties lead to simple proofs of the facts that Gaussian inputs achieve both the secrecy capacity of scalar Gaussian wiretap channels and the capacity of scalar Gaussian broadcast channels, as well as a simple proof of the entropy power inequality in the special case where one of the variables is Gaussian.

Dimensional behaviour of entropy and information

We develop an information-theoretic perspective on some questions in convex geometry, providing for instance a new equipartition property for log-concave probability measures, some Gaussian comparison results for log-concave measures, an entropic formulation of the hyperplane conjecture, and a new reverse entropy power inequality for log-concave measures analogous to V. Milman's reverse Brunn–Minkowski inequality.

Thinning, Entropy, and the Law of Thin Numbers

Renyi's "thinning" operation on a discrete random variable is a natural discrete analog of the scaling operation for continuous random variables. The properties of thinning are investigated in an information-theoretic context, especially in connection with information-theoretic inequalities related to Poisson approximation results. The classical Binomial-to-Poisson convergence (sometimes referred to as the "law of small numbers" is seen to be a special case of a thinning limit theorem for convolutions of discrete distributions. A rate of convergence is provided for this limit, and nonasymptotic bounds are also established. This development parallels, in part, the development of Gaussian inequalities leading to the information-theoretic version of the central limit theorem. In particular, a "thinning Markov chain" is introduced, and it is shown to play a role analogous to that of the Ornstein-Uhlenbeck process in connection to the entropy power inequality.

On the Sum Rate of Gaussian Multiterminal Source Coding: New Proofs and Results

We show that the lower bound on the sum rate of the direct and indirect Gaussian multiterminal source coding problems can be derived in a unified manner by exploiting the semidefinite partial order of the distortion covariance matrices associated with the minimum mean squared error (MMSE) estimation and the so-called reduced optimal linear estimation, through which an intimate connection between the lower bound and the Berger-Tung upper bound is revealed. We give a new proof of the minimum sum rate of the indirect Gaussian multiterminal source coding problem (i.e., the Gaussian CEO problem). For the direct Gaussian multiterminal source coding problem, we derive a general lower bound on the sum rate and establish a set of sufficient conditions under which the lower bound coincides with the Berger-Tung upper bound. We show that the sufficient conditions are satisfied for a class of sources and distortion constraints; in particular, they hold for arbitrary positive definite source covariance matrices in the high-resolution regime. In contrast with the existing proofs, the new method does not rely on Shannon's entropy power inequality.

Monotonicity, Thinning, and Discrete Versions of the Entropy Power Inequality

We consider the entropy of sums of independent discrete random variables, in analogy with Shannon's Entropy Power Inequality, where equality holds for normals. In our case, infinite divisibility suggests that equality should hold for Poisson variables. We show that some natural analogues of the Entropy Power Inequality do not in fact hold, but propose an alternative formulation which does always hold. The key to many proofs of Shannon's Entropy Power Inequality is the behaviour of entropy on scaling of continuous random variables. We believe that R\'{e}nyi's operation of thinning discrete random variables plays a similar role to scaling, and give a sharp bound on how the entropy of ultra log-concave random variables behaves on thinning. In the spirit of the monotonicity results established by Artstein, Ball, Barthe and Naor, we prove a stronger version of concavity of entropy, which implies a strengthened form of our discrete Entropy Power Inequality.

A Vector Generalization of Costa's Entropy-Power Inequality With Applications

This paper considers an entropy-power inequality (EPI) of Costa and presents a natural vector generalization with a real positive semidefinite matrix parameter. The new inequality is proved using a perturbation approach via a fundamental relationship between the derivative of mutual information and the minimum mean-square error (MMSE) estimate in linear vector Gaussian channels. As an application, a new extremal entropy inequality is derived from the generalized Costa EPI and then used to establish the secrecy capacity regions of the degraded vector Gaussian broadcast channel with layered confidential messages.

Distributed MIMO Receiver—Achievable Rates and Upper Bounds

A multiple-input multiple-output (MIMO) system with a distributed receiver is considered. The system consists of a nomadic transmitter with several antennas, whose signal is received by multiple agents, exhibiting independent channel gains and an additive circular-symmetric Gaussian noise. In the nomadic regime, we assume that the agents do not have any decoding ability. These agents process their channel observations and forward them to the final destination through unidirectional lossless links with a fixed capacity. We propose new achievable rates based on elementary compression and on Wyner-Ziv (WZ)or chief executive officer (CEO) processing, for both fast-fading and block-fading channels, as well as for general discrete channels. The simpler two agents scheme is solved, up to an implicit equation with a single variable. Limiting the nomadic transmitter to circular-symmetric Gaussian signaling, new upper bounds are derived, based on the vector version of the entropy power inequality. Several asymptotic settings are analyzed. In addition, the upper bounds are analytically shown to be tight for several examples, while numerical calculations reveal a rather small gap in a finite 2 times 2 setting. The advantage of the WZ approach over elementary compression is shown, where only the former can achieve the optimal diversity-multiplexing tradeoff (DMT).

Hessian and Concavity of Mutual Information, Differential Entropy, and Entropy Power in Linear Vector Gaussian Channels

Within the framework of linear vector Gaussian channels with arbitrary signaling, the Jacobian of the minimum mean square error and Fisher information matrices with respect to arbitrary parameters of the system are calculated in this paper. Capitalizing on prior research where the minimum mean square error and Fisher information matrices were linked to information-theoretic quantities through differentiation, the Hessian of the mutual information and the entropy are derived. These expressions are then used to assess the concavity properties of mutual information and entropy under different channel conditions and also to derive a multivariate version of an entropy power inequality due to Costa.

REMARKS ON A SEMICIRCULAR PERTURBATION OF THE FREE FISHER INFORMATION

In this paper, we shall give the representation of a semicircular perturbation of the free Fisher information [Formula: see text] by the mean of the conditional variance [Formula: see text], where S is a standard semicircular element freely independent of X, and ε > 0. Using this representation, we will give alternative proofs of the free Fisher information inequality, in which the free analogue of Stam's inequality can be obtained as a special case, and of the free entropy power inequality in an infinitesimal approach to the free entropy.

Generalized Entropy Power Inequalities and Monotonicity Properties of Information

New families of Fisher information and entropy power inequalities for sums of independent random variables are presented. These inequalities relate the information in the sum of n independent random variables to the information contained in sums over subsets of the random variables, for an arbitrary collection of subsets. As a consequence, a simple proof of the monotonicity of information in central limit theorems is obtained, both in the setting of independent and identically distributed (i.i.d.) summands as well as in the more general setting of independent summands with variance-standardized sums.

An Extremal Inequality Motivated by Multiterminal Information-Theoretic Problems

We prove a new extremal inequality, motivated by the vector Gaussian broadcast channel and the distributed source coding with a single quadratic distortion constraint problems. As a corollary, this inequality yields a generalization of the classical entropy-power inequality (EPI). As another corollary, this inequality sheds insight into maximizing the differential entropy of the sum of two dependent random variables

Information Theoretic Proofs of Entropy Power Inequalities

While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai and Verd\'u used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power.

Quantum broadcasting problem in classical low-power signal processing

We prove a no-broadcasting theorem for the Holevo information of a noncommuting ensemble stating that no operation can generate a bipartite ensemble such that both copies have the same information as the original. We argue that upper bounds on the average information over both copies imply lower bounds on the quantum capacity required to send the ensemble without information loss. This is because a channel with zero quantum capacity has a unitary extension transferring at least as much information to its environment as it transfers to the output. For an ensemble being the time orbit of a pure state under a Hamiltonian evolution, we derive such a bound on the required quantum capacity in terms of properties of the input and output energy distribution. Moreover, we discuss relations between the broadcasting problem and entropy power inequalities. The broadcasting problem arises when a signal should be transmitted by a time-invariant device such that the outgoing signal has the same timing information as the incoming signal had. Based on previous results we argue that this establishes a link between quantum information theory and the theory of low power computing because the loss of timing information implies loss of free energy.

Some results concerning maximum Rényi entropy distributions

We consider the Student- t and Student- r distributions, which maximise Rényi entropy under a covariance condition. We show that they have information-theoretic properties which mirror those of the Gaussian distributions, which maximise Shannon entropy under the same condition. We introduce a convolution which preserves the Rényi maximising family, and show that the Rényi maximisers are the case of equality in a version of the Entropy Power Inequality. Further, we show that the Rényi maximisers satisfy a version of the heat equation, motivating the definition of a generalised Fisher information.

On Divergence-Power Inequalities

Expressions for (EPI Shannon type) Divergence-Power Inequalities (DPI) in two cases (time-discrete and band-limited time-continuous) of stationary random processes are given. The new expressions connect the divergence rate of the sum of independent processes, the individual divergence rate of each process, and their power spectral densities. All divergences are between a process and a Gaussian process with same second order statistics, and are assumed to be finite. A new proof of the Shannon entropy-power inequality EPI, based on the relationship between divergence and causal minimum mean-square error (CMMSE) in Gaussian channels with large signal-to-noise ratio, is also shown.

The Capacity Region of the Gaussian Multiple-Input Multiple-Output Broadcast Channel

The Gaussian multiple-input multiple-output (MIMO) broadcast channel (BC) is considered. The dirty-paper coding (DPC) rate region is shown to coincide with the capacity region. To that end, a new notion of an enhanced broadcast channel is introduced and is used jointly with the entropy power inequality, to show that a superposition of Gaussian codes is optimal for the degraded vector broadcast channel and that DPC is optimal for the nondegraded case. Furthermore, the capacity region is characterized under a wide range of input constraints, accounting, as special cases, for the total power and the per-antenna power constraints

A simple proof of the entropy-power inequality

This correspondence gives a simple proof of Shannon's entropy-power inequality (EPI) using the relationship between mutual information and minimum mean-square error (MMSE) in Gaussian channels.

A Conditional Entropy Power Inequality for Dependent Variables

We provide a condition under which a version of Shannon's entropy power inequality will hold for dependent variables. We first provide a Fisher information inequality extending that found in the independent case. The key ingredients are a conditional expectation representation for the score function of a sum, and the de Bruijn identity which relates entropy and Fisher information.