Minimum Relative Entropy Research Articles

Relations between the barycentric and von Neumann entropies of a density matrix

Band and Park have criticized the von Neumann entropy, S N( ϱ) =− Tr ϱ In ϱ, of a density matrix (ϱ) as an inappropriate indicator of the uncertainty in ϱ. Rather, they proposed using an index based on expanding ϱ over all possible (pure and impure) states (D), not one founded on the terse (entropy- minimizing) spectral expansion of ϱ. However, a natural prior distribution over D is not available, while the uniform distribution over the pure states (P) is invariant under all unitary transformations. The minimum relative (barycentric) entropy, S b(ϱ), with respect to this distribution over P can be determined, using parametrizations of P for spin- 1 2 , spin-1, spin- 3 2 and two-photon systems. The relevance, in this regard, of certain theorems of Rothaus concerning the Laplace transforms of measures is discussed. The relation between S N(ϱ) and S b(ϱ) is established by a duality theorem of convex programming.

Approximation of Density Functions by Sequences of Exponential Families

1. Introduction. Consider the estimation of a probability density func- tion p(x) defined on a bounded interval. We approximate the logarithm of the density by a basis function expansion consisting of polynomials, splines or trigonometric series. The expansion yields a regular exponential family within which we estimate the density by the method of maximum likelihood. This method of density estimation arises by application of the principle of maxi- mum entropy or minimum relative entropy subject to empirical constraints. We show that if the logarithm of the density has r square-integrable deriva- tives, f IDr log p12 < ox, then the sequence of density estimators P converges to p in the sense of relative entropy (Kullback-Leibler fp log(p/ pn) at rate Opr(1/m2r + m/n) as m -X c and m2/n -* 0 in the spline and trigonometric cases and m3/n -O 0 in the polynomial case, where m is the dimension of the family and n is the sample size. Boundary conditions are assumed for the density in the trigonometric case. This convergence rate specializes to Op r(n-2r/(2r+ 1)) by setting m = nl/(2r+ 1) when the log-density is known to have degree of smoothness at least r. Analogous convergence results for the relative entropy are shown to hold in general, for any class of log-density functions and sequence of finite-dimensional linear spaces having L2 and L. approximation properties. The approximation of log-densities using polynomials has previously been considered by Neyman (1937) to define alternatives for goodness-of-fit tests, by Good (1963) as an application of the method of maximum entropy or minimum relative entropy, by Crain (1974, 1976a, b 1977) who demonstrates existence and consistency of the maximum likelihood estimator and by Mead and

Tail entropy approximations

A three parameter tail entropy approximation ( TEA) suitable for reliability and risk analysis is described and illustrated by examples. The TEA is determined directly from sample data. It combines the best features of tail approximation and entropy optimization for structural safety analysis. Estimation of the distributions of basic random variables from sample data is a critical step in the analysis of structural safety and other random phenomena. Standard estimation methods may be simple, but results are sensitive to the necessary but arbitrary assumptions about distribution type. The method of minimum relative entropy is more objective and less sensitive to arbitrary assumptions, but requires optimization procedures and some of the results are mathematically intractable. The TEA method is a tractable and accurate combination, useful when a tail approximation is appropriate.

MRE hierarchical decomposition of general queueing network models

The principle of Minimum Relative Entropy (MRE), given fully decomposable subset and aggregate mean queue length, utilisation and flow-balance constraints, is used in conjunction with asymptotic connections to infinite capacity queues, to derive new analytic approximations for the conditional and marginal state probabilities of single class general closed queueing network models (QNMs) in the context of a multilevel variable aggregation scheme. The concept of subparallelism is applied to preserve the flow conservation and a universal MRE hierarchical decomposition algorithm is proposed for the approximate analysis of arbitrary closed queueing networks with single server queues and general service-times. Heuristic criteria towards an optimal coupling of network's units at each level of aggregation are suggested. As an illustration, the MRE algorithm is implemented iteratively by using the Generalised Exponential (GE) distributional model to approximate the service and asymptotic flow processes in the network. This algorithm captures the exact solution of separable queueing networks, while for general queueing networks it compares favourably against exact solutions and known approximations.

On the Role of Lag Selection in Minimum‐Relative‐Entropy Spectral Estimation

The possibility of improving the quality of the minimum relative entropy spectral estimates by properly selecting the set of autocorrelation values is demonstrated. The study concentrates on two aspects: resolvability and accuracy of peak location. Several numerical examples are given.

Mean-field approximation minimizes relative entropy

We derive the mean-field approximation from the information-theoretic principle of minimum relative entropy instead of by minimizing Peierls’s inequality for the Weiss free energy of statistical physics theory. We show that information theory leads to our statistical mechanics procedure. As an example, we consider a problem in binary image restoration. We find that mean-field annealing compares favorably with the stochastic approach.

MINIMUM RELATIVE ENTROPY INVERSION OF 1D DATA WITH APPLICATIONS1

AbstractThe pioneering work of E. T. Jaynes in the field of Bayesian/Maximum Entropy methods has been successfully explored in a number of disciplines. The principle of maximum entropy (PME) is remarkably powerful and versatile and leads to results which are devoid of spurious structure. Minimum relative entropy (MRE) is a method which has all the important attributes of the maximum‐entropy (ME) approach with the advantage that prior information may be easily included. These ‘soft’ prior constraints play a fundamental role in the solution of underdetermined problems. The MRE approach, like ME, has achieved considerable success in the field of spectral analysis where the spectrum is estimated from incomplete autocorrelations. In this paper we apply the MRE philosophy to 1D inverse problems where the model is not necessarily positive, and thus we show that MRE is a general method of tackling linear, underdetermined, inverse problems. We illustrate our discussion with examples which deal with the famous die problem introduced by Jaynes, the question of aliasing, determination of interval velocities from stacking velocities and, finally, the universal problem of band‐limited extrapolation. It is found that the MRE solution for the interval velocities, when a uniform prior velocity is assumed, is exactly the Dix formulation which is generally used in the seismic industry.

Fractile constrained entropy estimation of distributions based on scarce data

Abstract Minimization of relative entropy under fractile constraints is used to select a probability model for a random variable of unknown type, given only a random sample of independent observations of the variable. The fractile constraints replace the conventional moment constraints. They are based on the sample rule, an exact combinatorial relationship. The method selects one distribution in any candidate set that conveys the minimum amount of information about the variable, over and above the information contained in the sample. This model is best among the candidate distributions in an objective sense. The optimum reference distribution also minimizes the minimum relative entropy value relative to the sample; however, it does not in general satisfy the sample rule. But the method also yields a unique posterior distribution that does satisfy the sample rule. Together with the optimum reference distribution, this posterior minimizes the relative entropy, and in this sense it. too, can be considered as...

A correspondence principle for relative entropy minimization

Relative entropy minimization has been proposed as an inference method for problems with information in the form of constraints on the underlying probability model. We provide a theoretical justification for this procedure through a correspondence principle. In particular, for a convex constraints set Λ, we show that as the number of trials increases, the empirical distribution constrained to lie within Λ and associated with a discrete probability distribution p will become arbitrarily close with high probability to the distribution that minimizes the relative entropy between p and Λ.

Minimizing the relative entropy in a face

For a separating state ρ of a C *-algebra A, we give a limit formula for the minimal relative entropy S(ρ, ·) in any face, as well as for the unique minimizer. In terms of this minimum, we define a superadditive function $$\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\rho } $$ on the faces of A. In the case of a W *-algebra and normal ρ, $$\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\rho } $$ can be considered as function on the projection lattice of an abelian W *-subalgebra, which is dominated by $$\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\smile}$}}{\rho } $$ , is given by a normal positive, but not necessarily normalized linear functional on A. This functional is the unique solution of a minimal entropy problem.

Minimum relative entropy enhancement of FFT spectra

The typical method for obtaining a power spectrum from values of the correlation function is through the FFT (fast Fourier transform). A novel approach to improving the FFT spectrum when it is corrupted by noise is presented. The method minimises a relative entropy functional involving spectral densities and requires knowledge only of the variance of the noise that is corrupting the FFT spectrum.

Reconstruction Of Transform-Coded Images By Entropy Methods

A new, universal approach to reconstructing transform-coded images is proposed. The method views the images as a probability mass function (pmf), allowing the retained coefficients of a transform (Karhunen-Loeve, discrete cosine, slant, etc.) to be thought of as averages of the basis functions over the pmf. This sets the stage for reconstructing the original images by using the maximum entropy principle (MEP) and the minimum relative entropy principle (MREP) with the retained coefficients as constraints in the extremizations. A formulation combining the two methods is also proposed, resulting in a reconstruction algorithm that is fast, proceeding in an iterative way using the estimate from each coefficient as a prior pmf for the next one via the MREP. The proposed approaches are illustrated with images compressed by discrete cosine transform coding, and the results are compared with standard reconstruction using the inverse discrete cosine transform.

A comparison of some rules for probabilistic reasoning

Generalized Bayesian conditionals and Dempster-Shafer's conditionals are considered as probabilistic kinematics which hold under different conditions. In particular, generalized Bayes can be applied whenever the available evidence allows to partition the frame of reference. It will be pointed out how, in this case, it is always possible to get a probability function by a belief function by means of minimum (relative) entropy kinematics.

An information theoretic interface for a stochastic model management system

Model management systems are intended to access, retrieve, and process models from a model base for a user in response to relatively simple commands and queries. For stochastic models a major problem is the transformation of problem specific information available to the decision maker into model parameters. The problem results from the various mixtures of incomplete, uncertain, and subjective data typical of stochastic environments. This paper proposes an interface for stochastic model management based on the information, theoretic principle of relative-entropy minimization. The Relative-Entropy Model Management System Interface (REMMSI) has the following key characteristics: (1) both subjective and objective data may be processed by the system, (2) parameter distributions are dynamically updated in the presence of new information, (3) inconsistent information provided by the user may be accepted and processed, and (4) the approach is applicable to a wide range of stochastic model management schemes.

On a relation between maximum likelihood classification and minimum relative-entropy classification (Corresp.)

Maximum likelihood (ML) and minimum relative-entropy (MRE) (minimum cross-entropy) classification of samples from an unknown probability density when the hypotheses comprise an exponential family are considered. It is shown that ML and MRE lead to the same classification nde, and the result is illustrated in terms of a method for estimating covariance matrices recently developed by Burg, Luenberger, and Wenger, MRE classification applies to the general case in which it cannot be assumed that the samples were generated by one of the hypothesis densities. The common use of ML in this case is technically incorrect, but the equivalence of MRE and ML provides a theoretical justification.

Optimal control of the decay of nonequilibrium statistical correlations

A quasithermodynamic interpretation of the stochastic control of irreversible thermodynamic diffusion processes is presented in which the drift is the control parameter. The joint entropy, considered as a function of the initial data is the negative of the optimal expected total cost, and the generalized Hamilton–Jacobi equation, which it satisfies, is the dynamic programming equation for the optimal stochastic drift control. The two cost functions, proposed by Yasue, are shown to differ by a stochastic gauge transformation and constitute equivalent variational problems. Yasue’s results are shown to apply in two limiting cases: the asymptotic time limit, where the nonequilibrium statistical correlations have worn off, or the weak noise limit, where random thermal fluctuations have a negligible importance. The asymptotic expected total cost is governed by a minimum relative entropy principle and the goodness of the thermodynamic drift control in the stochastic problem is determined.

Minimum cross-entropy spectral analysis

The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information, minimum relative entropy) is summarized, discussed, and applied to the classical problem of estimating power spectra given values of the autocorrelation function. This new method differs from previous methods in its explicit inclusion of a prior estimate of the power spectrum, and it reduces to maximum entropy spectral analysis as a special case. The prior estimate can be viewed as a means of shaping the spectral estimator. Cross-entropy minimization yields a family of shaped spectral estimators consistent with known autocorrelations. Results are derived in two equivalent ways: once by minimizing the cross-entropy of underlying probability densities, and once by arguments concerning the cross-entropy between the input and output of linear filters. Several example minimum cross-entropy spectra are included.