De Bruijn Identity Research Articles

De Bruijn identities in different Markovian channels

De Bruijn's identity in information theory states that if u is the solution of the heat equation, then the time derivative of the Shannon entropy for this solution is equal to the amount of Fisher information at u. In this article, we show how this identity changes if we replace the heat channel by the Fokker Planck, or passing from Fokker Planck to Ornstein-Uhlenbeck channels. Through these passages we investigate the different properties of these solutions. We exclusively dissect different properties of Ornstein-Uhlenbeck semigroup given by the Mehler formula expression. For more information see https://ejde.math.txstate.edu/Volumes/2023/12/abstr.html

Rate-Exponent Region for a Class of Distributed Hypothesis Testing Against Conditional Independence Problems

We study a class of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> -encoder hypothesis testing against conditional independence problems. Under the criterion that stipulates minimization of the Type II error subject to a (constant) upper bound <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon $ </tex-math></inline-formula> on the Type I error, we characterize the set of encoding rates and exponent for both discrete memoryless and memoryless vector Gaussian settings. For the DM setting, we provide a converse proof and show that it is achieved using the Quantize-Bin-Test scheme of Rahman and Wagner. For the memoryless vector Gaussian setting, we develop a tight outer bound by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. In particular, the result shows that for memoryless vector Gaussian sources the rate-exponent region is exhausted using the Quantize-Bin-Test scheme with Gaussian test channels; and there is no loss in performance caused by restricting the sensors’ encoders not to employ time sharing. Furthermore, we also study a variant of the problem in which the source, not necessarily Gaussian, has finite differential entropy and the sensors’ observations noises under the null hypothesis are Gaussian. For this model, our main result is an upper bound on the exponent-rate function. The bound is shown to mirror a corresponding explicit lower bound, except that the lower bound involves the source power (variance) whereas the upper bound has the source entropy power. Part of the utility of the established bound is for investigating asymptotic exponent/rates and losses incurred by distributed detection as function of the number of sensors.

Fluctuation and dissipation in memoryless open quantum evolutions

Von Neumann entropy rate for open quantum systems is, in general, written in terms of entropy production and entropy flow rates, encompassing the second law of thermodynamics. When the open-quantum-system evolution corresponds to a quantum dynamical semigroup, we find a decomposition of the infinitesimal generator of the dynamics, that allows to relate the von Neumann entropy rate with the divergence-based quantum Fisher information, at any time. Applied to quantum Gaussian channels that are dynamical semigroups, our decomposition leads to the quantum analog of the generalized classical de Bruijn identity, thus expressing the quantum fluctuation-dissipation relation in that kind of channels. Finally, from this perspective, we analyze how stationarity arises.

Skewed Jensen—Fisher Divergence and Its Bounds

A non-uniform (skewed) mixture of probability density functions occurs in various disciplines. One needs a measure of similarity to the respective constituents and its bounds. We introduce a skewed Jensen–Fisher divergence based on relative Fisher information, and provide some bounds in terms of the skewed Jensen–Shannon divergence and of the variational distance. The defined measure coincides with the definition from the skewed Jensen–Shannon divergence via the de Bruijn identity. Our results follow from applying the logarithmic Sobolev inequality and Poincaré inequality.

Cumulative Residual and Relative Cumulative Residual Fisher Information and Their Properties

In this work, we propose cumulative residual Fisher information and relative cumulative residual Fisher information measures and establish some of their properties. We first show that these cumulative Fisher measures can be expressed based on the hazard function. We then define extended versions of cumulative residual entropy and cumulative residual Fisher information measures based on the Jensen inequality. We also discuss connections between these information measures based on a new version of de Bruijn's identity for survival functions.

ϕ-Informational Measures: Some Results and Interrelations.

In this paper, we focus on extended informational measures based on a convex function : entropies, extended Fisher information, and generalized moments. Both the generalization of the Fisher information and the moments rely on the definition of an escort distribution linked to the (entropic) functional . We revisit the usual maximum entropy principle—more precisely its inverse problem, starting from the distribution and constraints, which leads to the introduction of state-dependent -entropies. Then, we examine interrelations between the extended informational measures and generalize relationships such the Cramér–Rao inequality and the de Bruijn identity in this broader context. In this particular framework, the maximum entropy distributions play a central role. Of course, all the results derived in the paper include the usual ones as special cases.

From Rényi Entropy Power to Information Scan of Quantum States.

In this paper, we generalize the notion of Shannon’s entropy power to the Rényi-entropy setting. With this, we propose generalizations of the de Bruijn identity, isoperimetric inequality, or Stam inequality. This framework not only allows for finding new estimation inequalities, but it also provides a convenient technical framework for the derivation of a one-parameter family of Rényi-entropy-power-based quantum-mechanical uncertainty relations. To illustrate the usefulness of the Rényi entropy power obtained, we show how the information probability distribution associated with a quantum state can be reconstructed in a process that is akin to quantum-state tomography. We illustrate the inner workings of this with the so-called “cat states”, which are of fundamental interest and practical use in schemes such as quantum metrology. Salient issues, including the extension of the notion of entropy power to Tsallis entropy and ensuing implications in estimation theory, are also briefly discussed.

Vector Gaussian CEO Problem Under Logarithmic Loss and Applications

In this paper, we study the vector Gaussian Chief Executive Officer (CEO) problem under logarithmic loss distortion measure. Specifically, ${K} \geq 2$ agents observe independently corrupted Gaussian noisy versions of a remote vector Gaussian source, and communicate independently with a decoder or CEO over rate-constrained noise-free links. The CEO also has its own Gaussian noisy observation of the source and wants to reconstruct the remote source to within some prescribed distortion level where the incurred distortion is measured under the logarithmic loss penalty criterion. We find an explicit characterization of the rate-distortion region of this model. The result can be seen as the counterpart to the vector Gaussian setting of that by Courtade-Weissman which provides the rate-distortion region of the model in the discrete memoryless setting. For the proof of this result, we obtain an outer bound by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. The approach is similar to Ekrem-Ulukus outer bounding technique for the vector Gaussian CEO problem under quadratic distortion measure, for which it was there found generally non-tight; but it is shown here to yield a complete characterization of the region for the case of logarithmic loss measure. Also, we show that Gaussian test channels with time-sharing exhaust the Berger-Tung inner bound, which is optimal. Furthermore, application of our results allows us to find the complete solutions of two related problems: a quadratic vector Gaussian CEO problem with determinant constraint and the vector Gaussian distributed Information Bottleneck problem. Finally, we develop Blahut-Arimoto type algorithms that allow to compute numerically the regions provided in this paper, for both discrete and Gaussian models. With the known relevance of the logarithmic loss fidelity measure in the context of learning and prediction, the proposed algorithms may find usefulness in a variety of applications where learning is performed distributively. We illustrate the efficiency of our algorithms through some numerical examples.

Symmetries of the quaternionic Ginibre ensemble

We establish a few properties of eigenvalues and eigenvectors of the quaternionic Ginibre ensemble (QGE), analogous to what is known in the complex Ginibre case (see [7, 11, 14]). We first recover a version of Kostlan’s theorem that was already at the heart of an argument by Rider [1], namely, that the set of the squared radii of the eigenvalues is distributed as a set of independent gamma variables. Our proof technique uses the De Bruijn identity and properties of Pfaffians; it also allows to prove that the high powers of these eigenvalues are independent. These results extend to any potential beyond the Gaussian case, as long as radial symmetry holds; this includes for instance truncations of quaternionic unitary matrices, products of quaternionic Ginibre matrices, and the quaternionic spherical ensemble. We then study the eigenvectors of quaternionic Ginibre matrices. Angles between eigenvectors and the matrix of overlaps both exhibit some specific features that can be compared to the complex case. In particular, we compute the distribution and the limit of the diagonal overlap associated to an eigenvalue that is conditioned to be at the origin. This complements a recent study of overlaps in quaternionic ensembles by Akemann, Förster and Kieburg [1, 2].

ENTROPY FLOW AND DE BRUIJN'S IDENTITY FOR A CLASS OF STOCHASTIC DIFFERENTIAL EQUATIONS DRIVEN BY FRACTIONAL BROWNIAN MOTION

AbstractMotivated by the classical De Bruijn's identity for the additive Gaussian noise channel, in this paper we consider a generalized setting where the channel is modelled via stochastic differential equations driven by fractional Brownian motion with Hurst parameter H ∈ (0, 1). We derive generalized De Bruijn's identity for Shannon entropy and Kullback–Leibler divergence by means of Itô's formula, and present two applications. In the first application we demonstrate its equivalence with Stein's identity for Gaussian distributions, while in the second application, we show that for H ∈ (0, 1/2], the entropy power is concave in time while for H ∈ (1/2, 1) it is convex in time when the initial distribution is Gaussian. Compared with the classical case of H = 1/2, the time parameter plays an interesting and significant role in the analysis of these quantities.

Generalization of the de Bruijn Identity to General &lt;inline-formula&gt; &lt;tex-math notation="LaTeX"&gt;$\phi$ &lt;/tex-math&gt; &lt;/inline-formula&gt;-Entropies and &lt;inline-formula&gt; &lt;tex-math notation="LaTeX"&gt;$\phi$ &lt;/tex-math&gt; &lt;/inline-formula&gt;-Fisher Informations

In this paper, we propose generalizations of the de Bruijn identity based on extensions of the Shannon entropy, Fisher information and their associated divergences or relative measures. The foundations of these generalizations are the $\phi $ -entropies and divergences of the Csiszar (or Salicru) class considered within a multidimensional context, including the one-dimensional case, and for several types of noisy channels characterized by a more general probability distribution beyond the well-known Gaussian noise. We found that the gradient and/or the Hessian of these entropies or divergences with respect to the noise parameter naturally give rise to generalized versions of the Fisher information or divergence, which are named the $\phi $ -Fisher information (divergence). The obtained identities can be viewed as further extensions of the classical de Bruijn identity. Analogously, it is shown that a similar relation holds between the $\phi $ -divergence and an extended mean-square error, named $\phi $ -mean square error, for the Gaussian channel.

Information Measures, Inequalities and Performance Bounds for Parameter Estimation in Impulsive Noise Environments

Recent studies found that many channels are affected by additive noise that is impulsive in nature and is best explained by heavy-tailed symmetric alpha-stable distributions. Dealing with impulsive noise environments comes with an added complexity with respect to the standard Gaussian environment: the alpha-stable probability density functions do not possess closed-form expressions except in few special cases. Furthermore, they have an infinite second moment and the “nice” Hilbert space structure of the space of random variables having a finite second moment is lost along with its tools and methodologies. This is indeed the case in estimation theory, where classical tools to quantify the performance of an estimator are tightly related to the assumption of having finite variance variables. In alpha-stable environments, expressions, such as the mean square error and the Cramer–Rao bound, are hence problematic. In this paper, we tackle the parameter-estimation problem in the impulsive noise environments and develop novel tools that are tailored to the alpha-stable and heavy-tailed noise environments, tools that coincide with the standard ones adopted in the Gaussian setup, namely, a generalized “power” measure and a generalized Fisher information. We generalize known information inequalities commonly used in the Gaussian context: the de Bruijn identity, the Fisher information inequality, the isoperimetric inequality for entropies and the Cramer–Rao bound. Additionally, we derive upper bounds on the differential entropy of independent sums having a stable component. Intermediately, the new power measure is used to shed some light on the additive alpha-stable noise channel capacity in a setup that generalizes the linear average power constrained additive white Gaussian noise channel. Our theoretical findings are paralleled with numerical evaluations of various quantities and bounds using developed MATLAB packages.

IT Formulae for Gamma Target: Mutual Information and Relative Entropy

In this paper, we introduce new Stein identities for gamma target distribution as well as a new non-linear channel specifically designed for gamma inputs. From these two ingredients, we derive an explicit and simple formula for the derivative of the input-output mutual information of this non-linear channel with respect to the channel quality parameter. This relation is reminiscent of the well-known link between the derivative of the input-output mutual information of additive Gaussian noise channel with respect to the signal-to-noise ratio and the minimum mean-square error. The proof relies on a rescaled version of De Bruijn identity for gamma target distribution together with a stochastic representation for the gamma specific Fisher information. Finally, we are able to derive precise bounds and asymptotics for the input-output mutual information of the non-linear channel with gamma inputs.

A stroll along the gamma

We provide the first in-depth study of the Laguerre interpolation scheme between an arbitrary probability measure and the gamma distribution. We propose new explicit representations for the Laguerre semigroup as well as a new intertwining relation. We use these results to prove a local De Bruijn identity which hold under minimal conditions. We obtain a new proof of the logarithmic Sobolev inequality for the gamma law with α≥1/2 as well as a new type of HSI inequality linking relative entropy, Stein discrepancy and standardized Fisher information for the gamma law with α≥1/2.

Extensions of the I-MMSE Relationship to Gaussian Channels With Feedback and Memory

Unveiling a fundamental link between information theory and estimation theory, the I-MMSE relationship by Guo et al., together with its numerous extensions, has great theoretical significance and various practical applications. On the other hand, its influences to date have been restricted to channels without feedback or memory, due to the absence of its extensions to such channels. In this paper, we propose the extensions of the I-MMSE relationship to discrete-time and continuous-time Gaussian channels with feedback and/or memory. Our approach is based on a very simple observation, which can be applied to other scenarios, such as a simple and direct proof of the classical de Bruijn's identity.

Extension of de Bruijn's identity to dependent non-Gaussian noise channels

Abstract De Bruijn's identity relates two important concepts in information theory: Fisher information and differential entropy. Unlike the common practice in the literature, in this paper we consider general additive non-Gaussian noise channels where more realistically, the input signal and additive noise are not independently distributed. It is shown that, for general dependent signal and noise, the first derivative of the differential entropy is directly related to the conditional mean estimate of the input. Then, by using Gaussian and Farlie–Gumbel–Morgenstern copulas, special versions of the result are given in the respective case of additive normally distributed noise. The previous result on independent Gaussian noise channels is included as a special case. Illustrative examples are also provided.

A DE BRUIJN'S IDENTITY FOR DEPENDENT RANDOM VARIABLES BASED ON COPULA THEORY

De Bruijn's identity shows a link between two fundamental concepts in information theory: entropy and Fisher information. In the literature, De Bruijn's identity has been stated under the assumption of independence between input signal and an additive noise. However, in the real world, the noise could be highly dependent on the main signal. The main aim of this paper is, firstly, to extend De bruijn's identity for signal-dependent noise channels and, secondly, to study how Stein and heat identities are related to De bruijn's identity. Thus, new versions of De Bruijn's identity are introduced in the case when input signal and additive noise are dependent and are jointly distributed according to Archimedean and Gaussian copulas. It is shown that in this generalized model, the derivatives of the differential entropy can be expressed in terms of a function of Fisher information. Our finding enfolds the conventional De Bruijn's identity as some remarks. Then, the equivalence among the new De Bruijn-type identity, Stein's identity and heat equation identity is established. The paper concludes with an application of the developed results in information theory.

An Outer Bound for the Vector Gaussian CEO Problem

We study the vector Gaussian CEO problem, where there are arbitrary number of agents, each having a noisy observation of a vector Gaussian source. The goal of the agents is to describe the source to a central unit, which wants to reconstruct the source within a given distortion. The rate-distortion region of the vector Gaussian CEO problem is unknown in general. Here, we provide an outer bound for the rate-distortion region of the vector Gaussian CEO problem. We obtain our outer bound by evaluating an outer bound for the multiterminal source coding problem by means of a technique relying on the de Bruijn identity and properties of the Fisher information. Next, we investigate the tightness of our outer bound. Although our outer bound is tight for certain cases, we show that our outer bound does not provide the exact rate-distortion region in general. To this end, we provide an example and show that the rate-distortion region is strictly contained in our outer bound for this example.

Information Theory in the mathematical Statistics

In the paper we present an outline of the information theory from the probabilistic and statistical point of view. Such a direction of the information theory has been intensively developed in recent decades and significantly influenced a progress in the statistical methodology. The aim of the article is to introduce the reader into these problems, provide some intuitions and acquaint with a specific information-theoretic approach to the mathematical statistics. The first part of the paper is devoted to brief and easy of approach introduction to the main notions of the information theory like entropy, relative entropy (Kullback-Leibler distance), information projection and Fisher information as well as presentation of their most important properties including de Bruijn's identity, Fisher information inequalities and entropy power inequalities. In the short second part we give applications of the notions and results from the first part to limit theorems of the probability theory such as the asymptotic equipartition property, the convergence of empirical measures in the entropy distance, large deviation principle with emphasis to Sanov theorem, the convergence of distributions of homogeneous Markov chains in the entropy distance and the central limit theorem. The main, last part of the article shows some most significant and important applications of the information theory to the mathematical statistics. We discuss connections of the maximum likelihood estimators with the information projections and the notion of sufficient statistic from the information-theoretic point of view. The problems of source coding, channel capacity and an amount of information provided by statistical experiments are presented in a statistical framework. Some attention is paid to the expansion of Clarke and Barron and its corollaries e.g. in density estimation. Next, applications of the information theory to hypothesis testing is discussed. We give the classical Stein's Lemma and its generalization to testing composite hypothesis obtained by Bahadur and show their connections with the asymptotic efficiency of statistical tests. Finally, we briefly mention the problem of information criteria in a model seletion including the most popular two-stage minimal description length criterion of Rissanen. The enclosed literature is limited only to papers and books which are referred to in the paper.

Some properties of generalized Fisher information in the context of nonextensive thermostatistics

We present two extended forms of Fisher information that fit well in the context of nonextensive thermostatistics. We show that there exists an interplay between these generalized Fisher information, the generalized q-Gaussian distributions and the q-entropies. The minimum of the generalized Fisher information among distributions with a fixed moment, or with a fixed q-entropy is attained, in both cases, by a generalized q-Gaussian distribution. This complements the fact that the q-Gaussians maximize the q-entropies subject to a moment constraint, and yields new variational characterizations of the generalizedq-Gaussians. We show that the generalized Fisher information naturally pop up in the expression of the time derivative of the q-entropies, for distributions satisfying a certain nonlinear heat equation. This result includes as a particular case the classical de Bruijn identity. Then we study further properties of the generalized Fisher information and of their minimization. We show that, though non additive, the generalized Fisher information of a combined system is upper bounded. In the case of mixing, we show that the generalized Fisher information is convex for q≥1. Finally, we show that the minimization of the generalized Fisher information subject to moment constraints satisfies a Legendre structure analog to the Legendre structure of thermodynamics.