Relative Entropy Distance Research Articles

Quantification of coarse-graining error in Langevin and overdamped Langevin dynamics

In molecular dynamics and sampling of high dimensional Gibbs measures coarse-graining is an important technique to reduce the dimensionality of the problem. We will study and quantify the coarse-graining error between the coarse-grained dynamics and an effective dynamics. The effective dynamics is a Markov process on the coarse-grained state space obtained by a closure procedure from the coarse-grained coefficients. We obtain error estimates both in relative entropy and Wasserstein distance, for both Langevin and overdamped Langevin dynamics. The approach allows for vectorial coarse-graining maps. Hereby, the quality of the chosen coarse-graining is measured by certain functional inequalities encoding the scale separation of the Gibbs measure. The method is based on error estimates between solutions of (kinetic) Fokker–Planck equations in terms of large-deviation rate functionals.

Minimum Rates of Approximate Sufficient Statistics

Given a sufficient statistic for a parametric family of distributions, one can estimate the parameter without access to the data. However, the memory or code size for storing the sufficient statistic may nonetheless still be prohibitive. Indeed, for n independent samples drawn from a k-nomial distribution with d = k - 1 degrees of freedom, the length of the code scales as d log n + O(1). In many applications, we may not have a useful notion of sufficient statistics (e.g., when the parametric family is not an exponential family), and we may also not need to reconstruct the generating distribution exactly. By adopting a Shannon-theoretic approach in which we allow a small error in estimating the generating distribution, we construct various approximate sufficient statistics and show that the code length can be reduced to (d/2) log n + O(1). We consider errors measured according to the relative entropy and variational distance criteria. For the code constructions, we leverage Rissanen's minimum description length principle, which yields a non-vanishing error measured according to the relative entropy. For the converse parts, we use Clarke and Barron's formula for the relative entropy of a parameterized distribution and the corresponding mixture distribution. However, this method only yields a weak converse for the variational distance. We develop new techniques to achieve vanishing errors, and we also prove strong converses. The latter means that even if the code is allowed to have a nonvanishing error, its length must still be at least (d/2) log n.

Low-dimensional manifolds for exact representation of open quantum systems

Weakly nonlinear degrees of freedom in dissipative quantum systems tend to localize near manifolds of quasi-classical states. We present a family of analytical and computational methods for deriving optimal unitary model transformations based on representations of finite dimensional Lie groups. The transformations are optimal in that they minimize the quantum relative entropy distance between a given state and the quasi-classical manifold. This naturally splits the description of quantum states into quasi-classical coordinates that specify the nearest quasi-classical state and a transformed quantum state that can be represented in fewer basis levels. We derive coupled equations of motion for the coordinates and the transformed state and demonstrate how this can be exploited for efficient numerical simulation. Our optimization objective naturally quantifies the non-classicality of states occurring in some given open system dynamics. This allows us to compare the intrinsic complexity of different open quantum systems.

Quantifying non-Gaussianity of quantum-state correlation

We consider how to quantify non-Gaussianity for the correlation of a bipartite quantum state by using various measures such as relative entropy and geometric distances. We first show that an intuitive approach, i.e., subtracting the correlation of a reference Gaussian state from that of a target non-Gaussian state, fails to yield a non-negative measure with monotonicity under local Gaussian channels. Our finding clearly manifests that quantum-state correlations generally have no Gaussian extremality. We therefore propose a different approach by introducing relevantly averaged states to address correlation. This enables us to define a non-Gaussianity measure based on, e.g., the trace-distance and the fidelity, fulfilling all requirements as a measure of non-Gaussian correlation. For the case of the fidelity-based measure, we also present readily computable lower bounds of non-Gaussian correlation.

$f$ -Divergence Inequalities

This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of $\chi^2$ divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including "reverse Pinsker inequalities," as well as on the $E_\gamma$ divergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of $f$-divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the R\'enyi divergence in terms of the relative information spectrum are derived, leading to bounds on the R\'enyi divergence in terms of either the variational distance or relative entropy.

Tight Uniform Continuity Bounds for Quantum Entropies: Conditional Entropy, Relative Entropy Distance and Energy Constraints

We present a bouquet of continuity bounds for quantum entropies, falling broadly into two classes: First, a tight analysis of the Alicki-Fannes continuity bounds for the conditional von Neumann entropy, reaching almost the best possible form that depends only on the system dimension and the trace distance of the states. Almost the same proof can be used to derive similar continuity bounds for the relative entropy distance from a convex set of states or positive operators. As applications we give new proofs, with tighter bounds, of the asymptotic continuity of the relative entropy of entanglement, $E_R$, and its regularization $E_R^\infty$, as well as of the entanglement of formation, $E_F$. Using a novel "quantum coupling" of density operators, which may be of independent interest, we extend the latter to an asymptotic continuity bound for the regularized entanglement of formation, aka entanglement cost, $E_C=E_F^\infty$. Second, analogous continuity bounds for the von Neumann entropy and conditional entropy in infinite dimensional systems under an energy constraint, most importantly systems of multiple quantum harmonic oscillators. While without an energy bound the entropy is discontinuous, it is well-known to be continuous on states of bounded energy. However, a quantitative statement to that effect seems not to have been known. Here, under some regularity assumptions on the Hamiltonian, we find that, quite intuitively, the Gibbs entropy at the given energy roughly takes the role of the Hilbert space dimension in the finite-dimensional Fannes inequality.

Estimation of low rank density matrices: Bounds in Schatten norms and other distances

Let $\mathcal{S}_{m}$ be the set of all $m\times m$ density matrices (Hermitian positively semi-definite matrices of unit trace). Consider a problem of estimation of an unknown density matrix $\rho\in\mathcal{S}_{m}$ based on outcomes of $n$ measurements of observables $X_{1},\dots,X_{n}\in\mathbb{H}_{m}$ ($\mathbb{H}_{m}$ being the space of $m\times m$ Hermitian matrices) for a quantum system identically prepared $n$ times in state $\rho.$ Outcomes $Y_{1},\dots,Y_{n}$ of such measurements could be described by a trace regression model in which $\mathbb{E}_{\rho}(Y_{j}|X_{j})={\rm tr}(\rho X_{j}),j=1,\dots,n.$ The design variables $X_{1},\dots,X_{n}$ are often sampled at random from the uniform distribution in an orthonormal basis $\{E_{1},\dots,E_{m^{2}}\}$ of $\mathbb{H}_{m}$ (such as Pauli basis). The goal is to estimate the unknown density matrix $\rho$ based on the data $(X_{1},Y_{1}),\dots,(X_{n},Y_{n}).$ Let \[\hat{Z}:=\frac{m^{2}}{n}\sum_{j=1}^{n}Y_{j}X_{j}\] and let $\check{\rho}$ be the projection of $\hat{Z}$ onto the convex set $\mathcal{S}_{m}$ of density matrices. It is shown that for estimator $\check{\rho}$ the minimax lower bounds in classes of low rank density matrices (established earlier) are attained up logarithmic factors for all Schatten $p$-norm distances, $p\in[1,\infty]$ and for Bures version of quantum Hellinger distance. Moreover, for a slightly modified version of estimator $\check{\rho}$ the same property holds also for quantum relative entropy (Kullback-Leibler) distance between density matrices.

Autonomous Fall Detection With Wearable Cameras by Using Relative Entropy Distance Measure

Timely, precise, and reliable detection of fall events is very important for systems monitoring activities of elderly people, especially the ones living independently. In this paper, we propose an autonomous fall detection system by taking a completely different view compared with existing vision-based activity monitoring systems and applying a reverse approach. In our system, in contrast with static sensors installed at fixed locations, the camera is worn by the subject, and thus, monitoring is not limited only to areas where the sensors are located and extends to wherever the subject may travel. Moreover, the camera provides a richer set of data and helps lower the false positive rates compared with accelerometer-only systems. We employ a modified version of the histograms of oriented gradients (HOG) approach together with the gradient local binary patterns (GLBP). It is shown that, with the same training set, the GLBP feature is more descriptive and discriminative than HOG, histograms of template, and semantic local binary patterns. Moreover, we autonomously compute a threshold, for the detection of fall events, from the training data based on relative entropy, which is a member of Ali-Silvey distance measures. Experiments are performed with ten different people and a total of around 300 associated fall events indoors and outdoors. Experimental results show that, with the autonomously computed threshold, the proposed method provides 93.78% and 89.8% accuracy for detecting falls with indoor and outdoor experiments, respectively.

Medical Image Watermarking Technique in the Application of E- diagnosis Using M-Ary Modulation

A robust and lossless ROI medical image watermarking technique using M-Ary modulation is proposed in this paper. The system's effectiveness is experimentally tested for two different medical image modalities of brain using various quality measures as payload, PSNR, MSE and SSIM. The proposed algorithm provides high robustness, higher PSNR and low relative entropy distance as a criterion for protection.

Production Task Queue Optimization Based on Multi-Attribute Evaluation for Complex Product Assembly Workshop.

The production task queue has a great significance for manufacturing resource allocation and scheduling decision. Man-made qualitative queue optimization method has a poor effect and makes the application difficult. A production task queue optimization method is proposed based on multi-attribute evaluation. According to the task attributes, the hierarchical multi-attribute model is established and the indicator quantization methods are given. To calculate the objective indicator weight, criteria importance through intercriteria correlation (CRITIC) is selected from three usual methods. To calculate the subjective indicator weight, BP neural network is used to determine the judge importance degree, and then the trapezoid fuzzy scale-rough AHP considering the judge importance degree is put forward. The balanced weight, which integrates the objective weight and the subjective weight, is calculated base on multi-weight contribution balance model. The technique for order preference by similarity to an ideal solution (TOPSIS) improved by replacing Euclidean distance with relative entropy distance is used to sequence the tasks and optimize the queue by the weighted indicator value. A case study is given to illustrate its correctness and feasibility.

Nonequilibrium thermodynamics and a fluctuation theorem for individual reaction steps in a chemical reaction network

We have introduced an approach to nonequilibrium thermodynamics of an open chemical reaction network in terms of the propensities of the individual elementary reactions and the corresponding reverse reactions. The method is a microscopic formulation of the dissipation function in terms of the relative entropy or Kullback-Leibler distance which is based on the analogy of phase space trajectory with the path of elementary reactions in a network of chemical process. We have introduced here a fluctuation theorem valid for each opposite pair of elementary reactions which is useful in determining the contribution of each sub-reaction on the nonequilibrium thermodynamics of overall reaction. The methodology is applied to an oligomeric enzyme kinetics at a chemiostatic condition that leads the reaction to a nonequilibrium steady state for which we have estimated how each step of the reaction is energy driven or entropy driven to contribute to the overall reaction.

Quantum Conditional Mutual Information, Reconstructed States, and State Redistribution.

We give two strengthenings of an inequality for the quantum conditional mutual information of a tripartite quantum state recently proved by Fawzi and Renner, connecting it with the ability to reconstruct the state from its bipartite reductions. Namely, we show that the conditional mutual information is an upper bound on the regularized relative entropy distance between the quantum state and its reconstructed version. It is also an upper bound for the measured relative entropy distance of the state to its reconstructed version. The main ingredient of the proof is the fact that the conditional mutual information is the optimal quantum communication rate in the task of state redistribution.

Sample-based evaluation of global probabilistic sensitivity measures

The evaluation of probabilistic sensitivity requires scalar measures quantifying the difference between probability density functions. The efficient estimation of two such alternative measures (relative entropy and Hellinger distance) is investigated within the setting of a global sensitivity analysis. This analysis requires evaluation of these measures with respect to different marginal densities, all connected to the same auxiliary joint density. Estimation of these densities is based on implementation of kernel density estimation, using information from samples from their joint distribution. These samples are further explored for improving the computational efficiency whereas theoretical concepts related to these two measures are addressed.

Information Divergence and Distance Measures for Quantum States

Both information divergence and distance are measures of closeness of two quantum states which are widely used in the theory of information processing and quantum cryptography. For example, the quantum relative entropy and trace distance are well known. Here we introduce a number of new quantum information divergence and distance measures into the literature and discuss their relations and properties. We also propose a method to analyze the properties and relations of various distance and pseudo-distance measures.

Non-Approximation Addictive Update and Multiplicative Update Online Classifier Algorithm

Since the online learning framework that make a compromise of the correctness and conservativeness is proposed by Kivinen and Warmuth,the framework have been referenced widely,but in gradient descent and exponentiated gradient algorithms proposed by Kivinen and Warmuth,the approximation step in the derivation of loss function of objection function lead to bad results.In this work,by means of duality theory of optimization,the novel non-approximation classifier algorithms based on square distance and relative entropy loss,relative entropy distance and relative entropy loss are proposed.Experimental results show that the proposed classifiers are always more accurate than the gradient descent and exponentiated gradient algorithms proposed by Kivinen and Warmuth in the real datasets.

Convexity of the image of a quadratic map via the relative entropy distance

Let \(\psi : {\mathbb R}^n \longrightarrow {\mathbb R}^k\) be a map defined by \(k\) positive definite quadratic forms on \({\mathbb R}^n\). We prove that the relative entropy (Kullback-Leibler) distance from the convex hull of the image of \(\psi \) to the image of \(\psi \) is bounded above by an absolute constant. More precisely, we prove that for every point \(a=\left( a_1, \ldots , a_k\right) \) in the convex hull of the image of \(\psi \) satisfying \(a_1 + \cdots + a_k=1\) there is a point \(b=\left( b_1, \ldots , b_k\right) \) in the image of \(\psi \) satisfying \(b_1 + \cdots + b_k =1\) and such that \(\sum _{i=1}^k a_i \ln \left( a_i/b_i\right) < 4.8\). Similarly, we prove that for any integer \(m\) one can choose a convex combination \(b\) of at most \(m\) points from the image of \(\psi \) such that \(\sum _{i=1}^k a_i \ln \left( a_i/b_i\right) < 17/\sqrt{m}\).

Quantifying causal influences

Many methods for causal inference generate directed acyclic graphs (DAGs) that formalize causal relations between $n$ variables. Given the joint distribution on all these variables, the DAG contains all information about how intervening on one variable changes the distribution of the other $n-1$ variables. However, quantifying the causal influence of one variable on another one remains a nontrivial question. Here we propose a set of natural, intuitive postulates that a measure of causal strength should satisfy. We then introduce a communication scenario, where edges in a DAG play the role of channels that can be locally corrupted by interventions. Causal strength is then the relative entropy distance between the old and the new distribution. Many other measures of causal strength have been proposed, including average causal effect, transfer entropy, directed information, and information flow. We explain how they fail to satisfy the postulates on simple DAGs of $\leq3$ nodes. Finally, we investigate the behavior of our measure on time-series, supporting our claims with experiments on simulated data.

Berry–Esseen bounds in the entropic central limit theorem

Berry–Esseen-type bounds for total variation and relative entropy distances to the normal law are established for the sums of non-i.i.d. random variables.

(QUANTUMNESS IN THE CONTEXT OF) RESOURCE THEORIES

We review the basic idea behind resource theories, where we quantify quantum resources by specifying a restricted class of operations. This divides the state space into various sets, including states which are free (because they can be created under the class of operations), and those which are a resource (because they cannot be). One can quantify the worth of the resource by the relative entropy distance to the set of free states, and under certain conditions, this is a unique measure which quantifies the rate of state to state transitions. The framework includes entanglement, asymmetry and purity theory. It also includes thermodynamics, which is a hybrid resource theory combining purity theory and asymmetry. Another hybrid resource theory which merges purity theory and entanglement can be used to study quantumness of correlations and discord, and we present quantumness in this more general framework of resource theories.

Outage Probability Under Channel Distribution Uncertainty

Outage probability and capacity of a class of block fading MIMO channels are considered under partial channel distribution information. Specifically, the channel or its distribution is not known but the latter is known to belong to a class of distributions where each member is within a certain distance (uncertainty) from a nominal distribution. Relative entropy is used as a measure of distance between distributions. Compound outage probability defined as min (over the transmitted signal distribution) -max (over the channel distribution class) outage probability is introduced and investigated. This generalizes the standard outage probability to the case of partial channel distribution information. Compound outage probability characterization (via 1-D convex optimization and in a closed form), its properties, and approximations are given. It is shown to have two-regime behavior: when the nominal outage probability decreases (e.g., by increasing the SNR), the compound outage first decreases linearly down to a certain threshold (related to the relative entropy distance; this is the nominal outage-dominated regime) and then only logarithmically (i.e., very slowly; this is the uncertainty-dominated regime) so that no significant further decrease is possible. This suggests the following design guideline: the outage probability is decreased by increasing the SNR or optimizing the transmitted signal distribution (both decrease nominal outage) in the first regime and by reducing the channel distribution uncertainty (e.g., via better estimation) in the second one. The compound outage depends on the relative entropy distance and the nominal outage only, all other details (nominal fading and noise distributions) being irrelevant. The transmit signal distribution optimized for the nominal channel distribution is shown to be also optimal for the whole class of distributions. The effect of swapping the distributions in relative entropy is investigated and an error floor effect is established. The compound outage probability under L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</sub> distance constraint is also investigated. The obtained results hold in full generality, i.e., for the general channel model with arbitrary nominal fading and noise distributions.