Minimum Relative Entropy Principle Research Articles

The Separability Problem in Molecular Quantum Systems: Information-Theoretic Framework for Atoms in Molecules.

Even though molecules are fundamentally quantum entities, the concept of a molecule retains certain classical attributes concerning its constituents. This includes the empirical separability of a molecule into its three-dimensional, rigid structure in Euclidean space, a framework often obtained through experimental methods like X-Ray crystallography. In this work, we delve into the mathematical implications of partitioning a molecule into its constituent parts using the widely recognized Atoms-In-Molecules (AIM) schemes, aiming to establish their validity within the framework of Information Theory concepts. We have uncovered information-theoretical justifications for employing some of the most prevalent AIM schemes in the field of Chemistry, including Hirshfeld (stockholder partitioning), Bader's (topological dissection), and the quantum approach (Hilbert's space definition). In the first approach we have applied the generalized principle of minimum relative entropy derived from the Sharma-Mittal two-parameter functional, avoiding the need for an arbitrary selection of reference promolecular atoms. Within the ambit of topological-information partitioning, we have demonstrated that the Fisher information of Bader's atoms conform to a comprehensive theory based on the Principle of Extreme Physical Information avoiding the need of employing the Schwinger's principle, which has been proven to be problematic. For the quantum approach we have presented information-theoretic justifications for conducting Löwdin symmetric transformations on the density matrix to form atomic Hilbert spaces generating orthonormal atomic orbitals with maximum occupancy for a given wavefunction.

Relative cluster entropy for power-law correlated sequences

We propose an information-theoretical measure, the relative cluster entropy \mathcal{D_C}[P\|Q]𝒟𝒞[P∥Q], to discriminate among cluster partitions characterised by probability distribution functions P and Q. The measure is illustrated with the clusters generated by pairs of fractional Brownian motions with Hurst exponents H_1H1 and H_2H2 respectively. For subdiffusive, normal and superdiffusive sequences, the relative entropy sensibly depends on the difference between H_1H1 and H_2H2. By using the minimum relative entropy principle, cluster sequences characterized by different correlation degrees are distinguished and the optimal Hurst exponent is selected. As a case study, real-world cluster partitions of market price series are compared to those obtained from fully uncorrelated sequences (simple Browniam motions) assumed as a model. The minimum relative cluster entropy yields optimal Hurst exponents H_1H1=0.55, H_1H1=0.57, and H_1H1=0.63 respectively for the prices of DJIA, S and P500, NASDAQ: a clear indication of non-markovianity. Finally, we derive the analytical expression of the relative cluster entropy and the outcomes are discussed for arbitrary pairs of power-laws probability distribution functions of continuous random variables.

On assigning probabilities to new hypotheses

• Knowledge mining on varying sets of compound hypotheses is inspected. • Bayesian learning is equipped with technique allowing to extend the set of hypotheses. • The technique copes with learning with less data than the amount of parameters. • The use of data forecasters as a knowledge source in parameter estimation is refined. • A rich set of interesting related research and application problems is presented. The paper proposes the way how to assign a proper prior probability to a new, generally compound, hypothesis. To this purpose, it uses the minimum relative-entropy principle and a forecaster-based knowledge transfer. Methodologically, it opens a way towards enriching the standard Bayesian framework by the possibility to extend the set of models during learning without the need to restart. The presented use scenarios concern: (a) creating new hypotheses, (b) learning problems with an insufficient amount of data, and (c) sequential Monte Carlo estimation. They indicate a strong application potential of the proposed technique. Related interesting open research problems are listed.

The Evaluation of Effectiveness for the Collaborative Combat of an Unmanned Aerial Vehicle Based on Grey Minimum Entropy

The main points to the evaluation of effectiveness for the collaborative combat of the unmanned aerial vehicle (UAV) lie within the construction of a reasonable indicator system and an accurate contribution model. As for point one, this article introduces a new method combining the Delphi consulting method and the principal component analysis method to avoid the underlying subjective and time-consuming defects of the existing methods. As for another point, a weighting method is adopted combining the subjective and objective parameters to minimize the errors caused by a single entity. Firstly, the modified grey relational degree analysis method is used to obtain the subjective weight, which can reduce the influence of the extreme values and outliers by enhancing the selection process of the reference sequence. Secondly, this paper adopts the weight of minimum entropy weight method to obtain the objective weight; it can avoid the information loss caused by the original method, which only determines the weight based on the frequency of each element present in the effective combination. At last, the principle of minimum relative entropy is adopted to obtain a more reasonable weight distribution coefficient. The simulation experiments established the rationality and effectiveness of the proposed method.

Multi-attribute group decision making method based on prospect theory under hesitant probabilistic fuzzy environment

The hesitant fuzzy element (HFE), as an extension of fuzzy numbers, has received much attention and the occurrence probabilities of each membership in the HFEs are assumed to be equal, i.e., the decision makers (DMs) share the same preference for each membership. Anecdotal observations suggest otherwise. To address this issue, the hesitant probabilistic fuzzy set (HPFS) is proposed, which is more suitable for depicting the DMs’ preferences in decision making. Research reports that psychological behavior matters in the decision-making process. Consequently, this paper proposes a multi-attribute group decision making method under a hesitant probabilistic fuzzy environment that considers the DMs’ psychological state in the decision-making process. A weights determination model is constructed, incorporating objective decision-making information and the subjective preferences of the DMs based on a score function and the minimum relative entropy principle. Finally, an established case example is applied to validate the applicability of the proposed method, and comparative analysis is also completed to verify the validity of the proposed method.

Relative Entropy-Based Density Selection in Particle Filtering for Load Demand Forecast

Particle filtering (PF) is an eminent simulation-based state estimation technique, which is capable of handling massive data sets and diverse external factors. Here, an effective selection of the importance density plays a pivotal role in performances of PFs by not only preventing degeneracy problems at early stages of process, but also taking both the transition prior and likelihood into account when the likelihood appears in the tail of the prior. To this end, we propose a novel importance density selection scheme for PF based on the minimum relative entropy principle. Theoretical derivation of the proposed scheme is presented, and its effectiveness is evaluated against various sampling schemes that exist in the literature using synthetic experiments. Finally, the performance of the proposed minimum relative entropy-based density selection scheme is successfully demonstrated for short-term electricity demand forecasting of a company located in Miami, FL, USA.

A Minimum Relative Entropy Principle for Learning and Acting

This paper proposes a method to construct an adaptive agent that is universal with respect to a given class of experts, where each expert is designed specifically for a particular environment. This adaptive control problem is formalized as the problem of minimizing the relative entropy of the adaptive agent from the expert that is most suitable for the unknown environment. If the agent is a passive observer, then the optimal solution is the well-known Bayesian predictor. However, if the agent is active, then its past actions need to be treated as causal interventions on the I/O stream rather than normal probability conditions. Here it is shown that the solution to this new variational problem is given by a stochastic controller called the Bayesian control rule, which implements adaptive behavior as a mixture of experts. Furthermore, it is shown that under mild assumptions, the Bayesian control rule converges to the control law of the most suitable expert.

A minimum relative entropy principle for learning and acting

This paper proposes a method to construct an adaptive agent that is universal with respect to a given class of experts, where each expert is designed specifically for a particular environment. This...

Bayesian generalized probability calculus for density matrices

One of the main concepts in quantum physics is a density matrix, which is a symmetric positive definite matrix of trace one. Finite probability distributions can be seen as a special case when the density matrix is restricted to be diagonal. We develop a probability calculus based on these more general distributions that includes definitions of joints, conditionals and formulas that relate these, including analogs of the Theorem of Total Probability and various Bayes rules for the calculation of posterior density matrices. The resulting calculus parallels the familiar “conventional” probability calculus and always retains the latter as a special case when all matrices are diagonal. We motivate both the conventional and the generalized Bayes rule with a minimum relative entropy principle, where the Kullbach-Leibler version gives the conventional Bayes rule and Umegaki’s quantum relative entropy the new Bayes rule for density matrices. Whereas the conventional Bayesian methods maintain uncertainty about which model has the highest data likelihood, the generalization maintains uncertainty about which unit direction has the largest variance. Surprisingly the bounds also generalize: as in the conventional setting we upper bound the negative log likelihood of the data by the negative log likelihood of the MAP estimator.

Reservoir-parameter identification using minimum relative entropy-based Bayesian inversion of seismic AVA and marine CSEM data

A stochastic joint-inversion approach for estimating reservoir-fluid saturations and porosity is proposed. The approach couples seismic amplitude variation with angle (AVA) and marine controlled-source electromagnetic (CSEM) forward models into a Bayesian framework, which allows for integration of complementary information. To obtain minimally subjective prior probabilities required for the Bayesian approach, the principle of minimum relative entropy (MRE) is employed. Instead of single-value estimates provided by deterministic methods, the approach gives a probability distribution for any unknown parameter of interest, such as reservoir-fluid saturations or porosity at various locations. The distribution means, modes, and confidence intervals can be calculated, providing a more complete understanding of the uncertainty in the parameter estimates. The approach is demonstrated using synthetic and field data sets. Results show that joint inversion using seismic and EM data gives better estimates of reservoir parameters than estimates from either geophysical data set used in isolation. Moreover, a more informative prior leads to much narrower predictive intervals of the target parameters, with mean values of the posterior distributions closer to logged values.

On minimum relative entropy concepts and prior compatibility issues in vadose zone inverse and forward modeling

A comprehensive stochastic approach for forward modeling of the state variables in the vadose zone as well as for inverse modeling of the hydraulic parameters is proposed and applied to a field site in Napa Valley, California. The approach combines the Richards equation and the van Genuchten–Mualem soil models, with initial and boundary conditions provided by geophysical and meteorological measurements, into a Bayesian formalism, which allows integration between different complementary information sources. A major challenge in such studies is the integration between different types of data, of different quality and resolution, and with prior information. Accurate modeling of prior information is challenging because of the need to avoid subjective judgment with regard to the quality of the prior information and its significance. To obtain minimally subjective prior probabilities required for the Bayesian approach, we employ the principle of minimum relative entropy (MRE). It can often be the case that information that is considered a suitable prior may, in fact, become incompatible with field observations as more observations become available. Our study considers this possibility and explores possible indications for prior incompatibility. We test our forward and inverse modeling approaches using field data. The results indicate that our approach is consistent, in the sense that as additional data are introduced, the simulated soil moisture content profile better matches the observed profile and the predictive intervals become much narrower. The narrower intervals of the soil moisture predictions indicate the reduction of uncertainties associated with the model parameters.

Arbitrary open queueing networks with server vacation periods and blocking

A cost-effective decomposition algorithm is proposed for the approximate analysis of arbitrary first-come-first-served (FCFS) open queueing network models (QNMs) with Generalised Exponential (GE) external interarrival time and service time distributions, multiple vacation periods, exhaustive service scheme and repetitive service blocking of random destination (RS-RD). The algorithm is based on a new product-form approximation suggested by the information theoretic principle of Minimum Relative Entropy (MRE), subject to marginal mean value constraints and a prior joint distribution estimate of the corresponding open QNM with infinite capacity. The censored GEyGEy1yN queue with exponential server vacations is solved via relative entropy minimisation and in conjunction with associated effective flow streams (departure, splitting, merging), plays the role of an efficient building block in the solution process. An exhaustive validation study against simulation is included to illustrate the credibility of the MRE results.

Practical Probabilistic Ground‐Water Modeling

AbstractThe major obstruction to the effective use of probabilistic models is the determination of the statistical properties of unknown model parameters. In this paper we use the principle of minimum relative entropy (MRE) to determine the prior pdf, p(m) of a set of model parameters, (m) based on limited information. The pdf is of the form of a multivariate truncated exponential distribution. In this paper we use p(m) in Monte Carlo simulations to provide expected values in field variables such as drawdowns, pumping rates, and confidence limits. The examples presented illustrate some dangers associated with the practice in probabilistic modeling of assigning Gaussian pdf's as priors. First, such an assumption for the input parameters actually injects more information into the problem than may actually exist, whether consciously or unconsciously. This fact is born out by comparison with minimum relative entropy theory. Second, the output parameters as suggested from the Monte Carlo analysis cannot be assumed to be Gaussian distributed even when the prior pdf is in Gaussian form. In a practical setting, the significance of this result and the approximation of Gaussian form would depend on the cost, risk, and consequences of the decision being made.

A Justification for Applying the Principle of Minimum Relative Entropy to Information Integration Problems

SYNOPTIC ABSTRACTInformation integration activities typically occur in the early stages of decision making. The intent of these activities is to produce a clear picture of an uncertain situation by combining information from multiple sources. Bayesian inference may not be directly applicable at this stage of the decision process. This paper justifies the use of the principle of minimum relative entropy as an inference procedure for information integration. The justification provided establishes asymptotic equivalence between the minimum relative entropy result and the Bayes estimate with consistent information.

Approximate angular momentum projection using the maximum entropy principle

Maximum entropy techniques are used to approximate angular momentum projection for wave functions approximated by a Slater determinant of single-particle states. The projected energies are obtained from application of the minimum relative entropy principle.

Multilevel aggregation of central server models: a minimum relative entropy approach

The principle of minimum relative entropy (MRE) may be viewed as a method of inference for characterizing the form of an unknown but true probability distribution based on information expressed in terms of an estimate of the unknown distribution and known to exist true expected values. Given fully decomposable subset and aggregate mean queue length and flow-balance constraints under a hierarchical decomposition scheme, the principle is used, in conjunction with asymptotic connections to infinite capacity queues, to derive new closed-form approximations for the conditional and marginal state probabilities of a general central server model. It is shown that the MRE joint state probability, subject to the prior information available, is of product-form, and it reduces to the exact solution obtained if the network was separable. This kind of analysis provides a novel interpretation of the notion of multilevel aggregation and introduces the basis of a new methodology for the performance analysis of queueing ne...

MRE hierarchical decomposition of general queueing network models

The principle of Minimum Relative Entropy (MRE), given fully decomposable subset and aggregate mean queue length, utilisation and flow-balance constraints, is used in conjunction with asymptotic connections to infinite capacity queues, to derive new analytic approximations for the conditional and marginal state probabilities of single class general closed queueing network models (QNMs) in the context of a multilevel variable aggregation scheme. The concept of subparallelism is applied to preserve the flow conservation and a universal MRE hierarchical decomposition algorithm is proposed for the approximate analysis of arbitrary closed queueing networks with single server queues and general service-times. Heuristic criteria towards an optimal coupling of network's units at each level of aggregation are suggested. As an illustration, the MRE algorithm is implemented iteratively by using the Generalised Exponential (GE) distributional model to approximate the service and asymptotic flow processes in the network. This algorithm captures the exact solution of separable queueing networks, while for general queueing networks it compares favourably against exact solutions and known approximations.

Mean-field approximation minimizes relative entropy

We derive the mean-field approximation from the information-theoretic principle of minimum relative entropy instead of by minimizing Peierls’s inequality for the Weiss free energy of statistical physics theory. We show that information theory leads to our statistical mechanics procedure. As an example, we consider a problem in binary image restoration. We find that mean-field annealing compares favorably with the stochastic approach.

Reconstruction Of Transform-Coded Images By Entropy Methods

A new, universal approach to reconstructing transform-coded images is proposed. The method views the images as a probability mass function (pmf), allowing the retained coefficients of a transform (Karhunen-Loeve, discrete cosine, slant, etc.) to be thought of as averages of the basis functions over the pmf. This sets the stage for reconstructing the original images by using the maximum entropy principle (MEP) and the minimum relative entropy principle (MREP) with the retained coefficients as constraints in the extremizations. A formulation combining the two methods is also proposed, resulting in a reconstruction algorithm that is fast, proceeding in an iterative way using the estimate from each coefficient as a prior pmf for the next one via the MREP. The proposed approaches are illustrated with images compressed by discrete cosine transform coding, and the results are compared with standard reconstruction using the inverse discrete cosine transform.

Optimal control of the decay of nonequilibrium statistical correlations

A quasithermodynamic interpretation of the stochastic control of irreversible thermodynamic diffusion processes is presented in which the drift is the control parameter. The joint entropy, considered as a function of the initial data is the negative of the optimal expected total cost, and the generalized Hamilton–Jacobi equation, which it satisfies, is the dynamic programming equation for the optimal stochastic drift control. The two cost functions, proposed by Yasue, are shown to differ by a stochastic gauge transformation and constitute equivalent variational problems. Yasue’s results are shown to apply in two limiting cases: the asymptotic time limit, where the nonequilibrium statistical correlations have worn off, or the weak noise limit, where random thermal fluctuations have a negligible importance. The asymptotic expected total cost is governed by a minimum relative entropy principle and the goodness of the thermodynamic drift control in the stochastic problem is determined.