Pinsker's Inequality Research Articles

Uncertainty relation for symmetric Petz-Rényi relative entropy.

Holevo introduced a fidelity between quantum states that is symmetric and as effective as the trace distance in evaluating their similarity. This fidelity is bounded by a function of the trace distance, a relationship to which we will refer as Holevo's inequality. More broadly, Holevo's fidelity is part of a one-parameter family of symmetric Petz-Rényi relative entropies, which in turn satisfy a Pinsker's-like inequality with respect to the trace distance. Although Holevo's inequality is tight, Pinsker's inequality is loose for this family. We show that the symmetric Petz-Rényi relative entropies satisfy a tight inequality with respect to the trace distance, improving Pinsker's and reproducing Holevo's as a specific case. Additionally, we show how this result emerges from a symmetric Petz-Rényi uncertainty relation, a result that encompasses several relations in quantum and stochastic thermodynamics.

Convergence of Relative Entropy for Euler-Maruyama Scheme to Stochastic Differential Equations with Additive Noise.

For a family of stochastic differential equations driven by additive Gaussian noise, we study the asymptotic behaviors of its corresponding Euler-Maruyama scheme by deriving its convergence rate in terms of relative entropy. Our results for the convergence rate in terms of relative entropy complement the conventional ones in the strong and weak sense and induce some other properties of the Euler-Maruyama scheme. For example, the convergence in terms of the total variation distance can be implied by Pinsker's inequality directly. Moreover, when the drift is β(0<β<1)-Hölder continuous in the spatial variable, the convergence rate in terms of the weighted variation distance is also established. Both of these convergence results do not seem to be directly obtained from any other convergence results of the Euler-Maruyama scheme. The main tool this paper relies on is the Girsanov transform.

The Interplay between Error, Total Variation, Alpha-Entropy and Guessing: Fano and Pinsker Direct and Reverse Inequalities.

Using majorization theory via "Robin Hood" elementary operations, optimal lower and upper bounds are derived on Rényi and guessing entropies with respect to either error probability (yielding reverse-Fano and Fano inequalities) or total variation distance to the uniform (yielding reverse-Pinsker and Pinsker inequalities). This gives a general picture of how the notion of randomness can be measured in many areas of computer science.

Lower bound for entropy production rate in stochastic systems far from equilibrium

We show that the Schnakenberg's entropy production rate in a master equationis lower bounded by a function of the weight of the Markov graph, here defined as the sum of the absolute values of probability currents over the edges. The result is valid for time-dependent nonequilibrium entropy production rates. Moreover, in a general framework, we prove a theorem showing that the Kullback-Leibler divergence between distributions P(s) and P^{'}(s):=P(m(s)), where m is an involution, m(m(s))=s, is lower bounded by a function of the total variation of P and P^{'}, for any m. The bound is tight and it improves on Pinsker's inequality for this setup. This result illustrates a connection between nonequilibrium thermodynamics and graph theory with interesting applications.

The Differential Entropy of Mixtures: New Bounds and Applications

Mixture distributions are extensively used as a modeling tool in diverse areas from machine learning to communications engineering to physics, and obtaining bounds on the entropy of mixture distributions is of fundamental importance in many of these applications. This article provides sharp bounds on the entropy concavity deficit, which is the difference between the differential entropy of the mixture and the weighted sum of differential entropies of constituent components. Toward establishing lower and upper bounds on the concavity deficit, results that are of importance in their own right are obtained. In order to obtain nontrivial upper bounds, properties of the skew-divergence are developed and notions of “skew” <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$f$ </tex-math></inline-formula> -divergences are introduced; a reverse Pinsker inequality and a bound on Jensen-Shannon divergence are obtained along the way. Complementary lower bounds are derived with special attention paid to the case that corresponds to independent summation of a continuous and a discrete random variable. Several applications of the bounds are delineated, including to mutual information of additive noise channels, thermodynamics of computation, and functional inequalities.

Pinsker inequalities and related Monge-Ampere equations for log-concave functions

In this paper we further develop the theory of f-divergences for log-concave functions and their related inequalities. We establish Pinsker inequalities and new affine invariant entropy inequalities. We obtain new inequalities on functional affine surface area and lower and upper bounds for the Kullback-Leibler divergence in terms of functional affine surface area. The functional inequalities lead to new inequalities for L_p-affine surface areas for convex bodies.

Properties of a Generalized Divergence Related to Tsallis Generalized Divergence

In this paper, we investigate the partition inequality, joint convexity, and Pinsker's inequality, for a divergence that generalizes the Tsallis Relative Entropy and Kullback-Leibler divergence. The generalized divergence is defined in terms of a deformed exponential function, which replaces the Tsallis q-exponential. We also constructed a family of probability distributions related to the generalized divergence. We found necessary and sufficient conditions for the partition inequality to be satisfied. A sufficient condition for the joint convexity was established. We proved that the generalized divergence satisfies the partition inequality, and is jointly convex, if, and only if, it coincides with the Tsallis relative entropy. As an application of partition inequality, a criterion for the Pinsker's inequality was found.

Optimal Detection of UAV's Transmission With Beam Sweeping in Covert Wireless Networks

In this work, a detection strategy based on multiple antennas with beam sweeping is developed to detect the potential transmission of unmanned aerial vehicle (UAV) in wireless networks. Specifically, suspicious angle range where the UAV may present is divided into different sectors to potentially increase detection accuracy by using beamforming gain. We then develop the optimal detector and derive its detection error probability in a closed-form expression. We also utilize the Pinsker's inequality and Kullback-Leibler divergence to yield low-complex approximation for the detection error probability, based on which we obtain some significant insights on the detection performance. Our numerical results show that there exists an optimal number of sectors that can minimize the detection error probability in some scenarios (e.g., when the number of measurements is limited). Intuitively, this can be explained by the fact that there exists an optimal accuracy of the telescope used to find an object in the sky within limited time period.

A remainder term for Hölder’s inequality for matrices and quantum entropy inequalities

We prove a sharp remainder term for Holder’s inequality for traces as a consequence of the uniform convexity properties of the Schatten trace norms. We then show how this implies a novel family of Pinsker type bounds for the quantum Renyi entropy. Finally, we show how the sharp form of the usual quantum Pinsker inequality for relative entropy may be obtained as a fairly direct consequence of uniform convexity.

$f$ -Divergence Inequalities

This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of $\chi^2$ divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including "reverse Pinsker inequalities," as well as on the $E_\gamma$ divergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of $f$-divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the R\'enyi divergence in terms of the relative information spectrum are derived, leading to bounds on the R\'enyi divergence in terms of either the variational distance or relative entropy.

Minimum KL-Divergence on Complements of &lt;inline-formula&gt; &lt;tex-math notation="TeX"&gt;$L_{1}$ &lt;/tex-math&gt;&lt;/inline-formula&gt; Balls

Pinsker's widely used inequality upper-bounds the total variation distance ∥P - Q∥ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> in terms of the Kullback-Leibler divergence D(P∥Q). Although, in general, a bound in the reverse direction is impossible, in many applications the quantity of interest is actually D*(v, Q)-defined, for an arbitrary fixed Q, as the infimum of D(P∥Q) over all distributions P that are at least v-far away from Q in total variation. We show that D*(v, Q) ≤ Cv <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> + O(v <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ), where C = C(Q) = 1/2 for balanced distributions, thereby providing a kind of reverse Pinsker inequality. Some of the structural results obtained in the course of the proof may be of independent interest. An application to large deviations is given.

Remainder terms for some quantum entropy inequalities

We consider three von Neumann entropy inequalities: subadditivity; Pinsker's inequality for relative entropy; and the monotonicity of relative entropy. For these we state conditions for equality, and we prove some new error bounds away from equality, including an improved version of Pinsker's inequality.

Relative Entropy and Squashed Entanglement

We are interested in the properties and relations of entanglement measures. Especially, we focus on the squashed entanglement and relative entropy of entanglement, as well as their analogues and variants. Our first result is a monogamy-like inequality involving the relative entropy of entanglement and its one-way LOCC variant. The proof is accomplished by exploring the properties of relative entropy in the context of hypothesis testing via one-way LOCC operations, and by making use of an argument resembling that by Piani on the faithfulness of regularized relative entropy of entanglement. Following this, we obtain a commensurate and faithful lower bound for squashed entanglement, in the form of one-way LOCC relative entropy of entanglement. This gives a strengthening to the strong subadditivity of von Neumann entropy. Our result improves the trace-distance-type bound derived in [Comm. Math. Phys., 306:805-830, 2011], where faithfulness of squashed entanglement was first proved. Applying Pinsker's inequality, we are able to recover the trace-distance-type bound, even with slightly better constant factor. However, the main improvement is that our new lower bound can be much larger than the old one and it is almost a genuine entanglement measure. We evaluate exactly the various relative entropy of entanglement under restricted measurement classes, for maximally entangled states. Then, by proving asymptotic continuity, we extend the exact evaluation to their regularized versions for all pure states. Finally, we consider comparisons and separations between some important entanglement measures and obtain several new results on these, too.

Correlation Distance and Bounds for Mutual Information

The correlation distance quantifies the statistical independence of two classical or quantum systems, via the distance from their joint state to the product of the marginal states. Tight lower bounds are given for the mutual information between pairs of two-valued classical variables and quantum qubits, in terms of the corresponding classical and quantum correlation distances. These bounds are stronger than the Pinsker inequality (and refinements thereof) for relative entropy. The classical lower bound may be used to quantify properties of statistical models that violate Bell inequalities. Entangled qubits can have a lower mutual information than can any two-valued classical variables having the same correlation distance. The qubit correlation distance also provides a direct entanglement criterion, related to the spin covariance matrix. Connections of results with classically-correlated quantum states are briefly discussed.

Local Pinsker Inequalities via Stein's Discrete Density Approach

Pinsker's inequality states that the relative entropy between two random variables X and Y dominates the square of the total variation distance between X and Y. In this paper, we introduce generalized Fisher information distances and prove that these also dominate the square of the total variation distance. To this end, we introduce a general discrete Stein operator for which we prove a useful covariance identity. We illustrate our approach with several examples. Whenever competitor inequalities are available in the literature, the constants in ours are at least as good, and, in several cases, better.

Lower Bounds for the Minimax Risk Using $f$-Divergences, and Applications

Lower bounds involving f-divergences between the underlying probability measures are proved for the minimax risk in estimation problems. Our proofs just use simple convexity facts. Special cases and straightforward corollaries of our bounds include well known inequalities for establishing minimax lower bounds such as Fano's inequality, Pinsker's inequality and inequalities based on global entropy conditions. Two applications are provided: a new minimax lower bound for the reconstruction of convex bodies from noisy support function measurements and a different proof of a recent minimax lower bound for the estimation of a covariance matrix.

Information, Divergence and Risk for Binary Experiments

We unify f-divergences, Bregman divergences, surrogate regret bounds, proper scoring rules, cost curves, ROC-curves and statistical information. We do this by systematically studying integral and variational representations of these objects and in so doing identify their representation primitives which all are related to cost-sensitive binary classification. As well as developing relationships between generative and discriminative views of learning, the new machinery leads to tight and more general surrogate regret bounds and generalised Pinsker inequalities relating f-divergences to variational divergence. The new viewpoint also illuminates existing algorithms: it provides a new derivation of Support Vector Machines in terms of divergences and relates maximum mean discrepancy to Fisher linear discriminants.

On divergences of finite measures and their applicability in statistics and information theory

Modifications of the classical φ-divergences D φ(μ, ν)=∈t q φ (p/q) dλ of finite measures μ, ν on a σ-finite measure space (𝒳, 𝒜, λ) with Radon–Nikodym densities p=dμ/dλ, q=dν/dλ are introduced by the formula 𝔇φ(μ, ν)=∈t q φ˜ (p/q) dλ using the nonnegative convex functions . Basic properties of the modified φ-divergences are investigated, such as the range of values, symmetry and a decomposition into local and global components. A general φ-divergence formula for right-censored observations illustrates the statistical applicability. The Pinsker inequality for finite measures and the generalized Ornstein distance of stationary random processes are among the illustrations of applicability in the information theory.

Intrinsic geometry on the class of probability densities and exponential families

We present a way of thinking of exponential families as geodesic surfaces in the class of positive functions considered as a (multiplicative) sub-group $G^+$ of the group $G$ of all invertible elements in the algebra $\mathcal{A}$ of all complex bounded functions defined on a measurable space. For that we have to study a natural geometry on that algebra. The class $\mathcal{D}$ of densities with respect to a given measure will happen to be representatives of equivalence classes defining a projective space in $\mathcal{A}$. The natural geometry is defined by an intrinsic group action which allows us to think of the class of positive, invertible functions $G^+$ as a homogeneous space. Also, the parallel transport in $G^+$ and $\mathcal{D}$ will be given by the original group action. Besides studying some relationships among these constructions, we examine some Riemannian geometries and provide a geometric interpretation of Pinsker's and other classical inequalities. Also we provide a geometric reinterpretation of some relationships between polynomial sequences of convolution type, probability distributions on $\mathbb{N}$ in terms of geodesics in the Banach space $\ell_1(\alpha)$.

On Pinsker's and Vajda's Type Inequalities for Csiszár's $f$-Divergences

We study conditions on $f$ under which an $f$-divergence $D_f$ will satisfy $D_f \geq c_f V^2$ or $D_f \geq c_{2,f} V^2 + c_{4,f} V^4$, where $V$ denotes variational distance and the coefficients $c_f$, $c_{2,f}$ and $c_{4,f}$ are {\em best possible}. As a consequence, we obtain lower bounds in terms of $V$ for many well known distance and divergence measures. For instance, let $D_{(\alpha)} (P,Q) = [\alpha (\alpha-1)]^{-1} [\int q^{\alpha} p^{1-\alpha} d \mu -1]$ and ${\cal I}_\alpha (P,Q) = (\alpha -1)^{-1} \log [\int p^\alpha q^{1-\alpha} d \mu]$ be respectively the {\em relative information of type} ($1-\alpha$) and {\em R\'{e}nyi's information gain of order} $\alpha$. We show that $D_{(\alpha)} \geq {1/2} V^2 + {1/72} (\alpha+1)(2-\alpha) V^4$ whenever $-1 \leq \alpha \leq 2$, $\alpha \not= 0,1$ and that ${\cal I}_{\alpha} = \frac{\alpha}{2} V^2 + {1/36} \alpha (1 + 5 \alpha - 5 \alpha^2) V^4$ for $0 < \alpha < 1$. Pinsker's inequality $D \geq {1/2} V^2$ and its extension $D \geq {1/2} V^2 + {1/36} V^4$ are special cases of each one of these.