Kullback-Leibler Cross Entropy Research Articles

BUAK-AIS: Efficient Bayesian Updating with Active learning Kriging-based Adaptive Importance Sampling

Bayesian updating provides a sound mathematical framework for probabilistic calibration as new information emerges. Bayesian Updating with Structural reliability methods (BUS) reformulates the acceptance domain in rejection sampling as a failure domain in reliability analysis, offering considerable potential for higher efficiency and accuracy. Kriging-based Monte Carlo Simulation has been studied to facilitate the application of BUS for problems with expensive-to-evaluate likelihood functions. Nevertheless, as the implementation of BUS often involves a rare event, the number of required Monte Carlo samples can become unaffordable. This gap is addressed here through Bayesian Updating with Active learning Kriging-based Adaptive Importance Sampling (BUAK-AIS). An importance sampling density based on Gaussian mixture distribution is introduced, and the discrepancy between the adopted and theoretically best sampling densities is measured through the Kullback–Leibler cross entropy. The proposed method includes an active learning framework that adaptively extends the training set and optimizes the parameters of the Gaussian mixture distribution based on the cross entropy and the current Kriging model. As BUS uses accepted samples to estimate the posterior distribution, the present work discusses the estimate for the first moment of the posterior distribution, and proposes a criterion to check the sufficiency of the number of accepted samples to guarantee robust estimations. A new stopping criterion is also developed by quantifying the error introduced by Kriging. Three numerical examples and an engineering application concerning model updating of cable-stayed bridges in the construction process are investigated, demonstrating the efficiency and accuracy of the proposed method.

Nonoptimality of the Maximum-Weight Dependence Tree in Classification

Half a century ago, Chow and Liu proved that the distribution of the first-order dependence tree that optimally approximates an arbitrary joint distribution in terms of Kullback–Leibler cross entropy is the distribution of the maximum-weight dependence tree (MWDT). In a $p$ -dimensional space, this is a tree with $p-1$ branches between pairs of variables with each branch having a weight equal to the mutual information between variables at both ends. Today, MWDTs have applications that are more important in classification than only being used to model first-order dependence structure among variables. Nevertheless, whether the MWDT is the optimal first-order tree in classification has been left unexplored so far. In this letter, we study the optimality of the MWDT structure from the stand-point of classification.

Recovering the Most Entropic Copulas from Preliminary Knowledge of Dependence

This paper provides a new approach to recover relative entropy measures of contemporaneous dependence from limited information by constructing the most entropic copula (MEC) and its canonical form, namely the most entropic canonical copula (MECC). The MECC can effectively be obtained by maximizing Shannon entropy to yield a proper copula such that known dependence structures of data (e.g., measures of association) are matched to their empirical counterparts. In fact the problem of maximizing the entropy of copulas is the dual to the problem of minimizing the Kullback-Leibler cross entropy (KLCE) of joint probability densities when the marginal probability densities are fixed. Our simulation study shows that the proposed MEC estimator can potentially outperform many other copula estimators in finite samples.

Sparse Antenna Array Optimization With the Cross-Entropy Method

The interest in sparse antenna arrays is growing, mainly due to cost concerns, array size limitations, etc. Formally, it can be shown that their design can be expressed as a constrained multidimensional nonlinear optimization problem. Generally, through lack of convex property, such a multiextrema problem is very tricky to solve by usual deterministic optimization methods. In this article, a recent stochastic approach, called Cross-Entropy method, is applied to the continuous constrained design problem. The method is able to construct a random sequence of solutions which converges probabilistically to the optimal or the near-optimal solution. Roughly speaking, it performs adaptive changes to probability density functions according to the Kullback-Leibler cross-entropy. The approach efficiency is illustrated in the design of a sparse antenna array with various requirements.

ADVANCES IN CROSS-ENTROPY METHODS

Abstract

Extension of HRTEM resolution by semi-blind deconvolution method and Gerchberg-Saxton algorithm: application to grain boundary and interface.

A generalized maximum entropy method coupled with Gerchberg-Saxton algorithm has been developed to extend the resolution from high-resolution TEM image(s) for weak objects. The Gerchberg-Saxton algorithm restores spatial resolution by operating real space and reciprocal space projections cyclically. In our methodology, a generalized maximum entropy method (Kullback-Leibler cross entropy) dealing with weak objects is used as a real space (P1) projection. After P1 projection, not only are the phases within the input spatial frequencies improved, but also the phases in the next higher frequencies are extrapolated. An example of semi-blind deconvolution (P1 project only) to improve the resolution in SiC twin boundary is shown. The nature of the bonding in this twin boundary is Si-C but it was rotated 180 degrees along the boundary normal. The optimum solution from P1 projection can be further improved by a P2 projection. The square roots of diffraction intensities from a diffraction pattern are then substituted to complete a cycle operation of the Gerchberg-Saxton algorithm. Application examples of Gerchberg-Saxton algorithm to solve the atomic structure of defects (2 x 1 interfacial reconstruction and dislocation) in NiSi2/Si interfaces will be shown also.

Penniless propagation in join trees

This paper presents non-random algorithms for approximate computation in Bayesian networks. They are based on the use of probability trees to represent probability potentials, using the Kullback-Leibler cross entropy as a measure of the error of the approximation. Different alternatives are presented and tested in several experiments with difficult propagation problems. The results show how it is possible to find good approximations in short time compared with Hugin algorithm. © 2000 John Wiley & Sons, Inc.

Penniless propagation in join trees

This paper presents non-random algorithms for approximate computation in Bayesian networks. They are based on the use of probability trees to represent probability potentials, using the Kullback-Leibler cross entropy as a measure of the error of the approximation. Different alternatives are presented and tested in several experiments with difficult propagation problems. The results show how it is possible to find good approximations in short time compared with Hugin algorithm. © 2000 John Wiley & Sons, Inc.

The Cross-Entropy Method for Combinatorial and Continuous Optimization

We present a new and fast method, called the cross-entropy method, for finding the optimal solution of combinatorial and continuous nonconvex optimization problems with convex bounded domains. To find the optimal solution we solve a sequence of simple auxiliary smooth optimization problems based on Kullback-Leibler cross-entropy, importance sampling, Markov chain and Boltzmann distribution. We use importance sampling as an important ingredient for adaptive adjustment of the temperature in the Boltzmann distribution and use Kullback-Leibler cross-entropy to find the optimal solution. In fact, we use the mode of a unimodal importance sampling distribution, like the mode of beta distribution, as an estimate of the optimal solution for continuous optimization and Markov chains approach for combinatorial optimization. In the later case we show almost surely convergence of our algorithm to the optimal solution. Supporting numerical results for both continuous and combinatorial optimization problems are given as well. Our empirical studies suggest that the cross-entropy method has polynomial in the size of the problem running time complexity.

Iterative algorithms for deblurring and deconvolution with constraints

The deblurring problem is that of recovering the function from (noisy) values . The discrete finite version of the problem is to solve the system of linear equations Hc=d for c, where H is a matrix and d and c are vectors. When the kernel is a function of the difference , the deblurring problem becomes a deconvolution problem. The use of iterative algorithms to effect deblurring subject to non-negativity constraints on c has been presented by Snyder et al for the case of non-negative kernel function h. In this paper we extend these algorithms to include upper and lower bounds on the entries of the desired solution. We show that any linear deblurring problem involving a real kernel h can be transformed into a linear deblurring problem involving a non-negative kernel. Therefore our algorithms apply to general deblurring and deconvolution problems. These algorithms converge to a solution of the system of equations y = Px, with , for , , satisfying the vector inequalities , whenever such a solution exists. When there is no solution satisfying the constraints the simultaneous versions converge to an approximate solution that minimizes a cost function related to the Kullback-Leibler cross-entropy and the Fermi-Dirac generalized entropy.

VLBI Image Deconvolution Based on Generalized Nonlinear Information Methods

AbstractA number of nonlinear image deconvolution procedures based on extremizing generalized Shannon entropy, Kullback-Leibler cross-entropy and Renyi information measures are suggested for VLBI mapping with minimal nonlinear distortions. A new approach to phaseless mapping urgent in the case of unreliable closure phases is proposed.

Cross-entropy minimization for refinement of Gaussian basis sets

Information theoretic techniques have been applied for the refinement of Gaussian basis sets. A refined distribution has been obtained by cross-entropy minimization starting from a near Hartree-Fock quality density distribution. For this purpose, the Kullback-Leibler cross-entropy, S[ϱ¦ϱ 0] = ∫ϱ( r )ln[ϱ( r )/ϱ 0( r )] d r , has been minimized subject to exact, theoretical or experimental, second moment constraints in position and momentum spaces. Here, ϱ 0 is the starting density distribution and ϱ is the corresponding refined one. The procedure has been applied to hydrogen, helium, lithium and beryllium atoms as test cases. Nearly all moments (−2 through 4), in coordinate as well as momentum spaces have improved over the original ones at an average worsening of the total energy by a mere 0.04%.