Minimization Of Kullback-Leibler Divergence Research Articles

Simple variational inference based on minimizing Kullback–Leibler divergence

Simple variational inference based on minimizing Kullback–Leibler divergence

A deep clustering framework integrating pairwise constraints and a VMF mixture model

<abstract><p>We presented a novel deep generative clustering model called Variational Deep Embedding based on Pairwise constraints and the Von Mises-Fisher mixture model (VDEPV). VDEPV consists of fully connected neural networks capable of learning latent representations from raw data and accurately predicting cluster assignments. Under the assumption of a genuinely non-informative prior, VDEPV adopted a von Mises-Fisher mixture model to depict the hyperspherical interpretation of the data. We defined and established pairwise constraints by employing a random sample mining strategy and applying data augmentation techniques. These constraints enhanced the compactness of intra-cluster samples in the spherical embedding space while improving inter-cluster samples' separability. By minimizing Kullback-Leibler divergence, we formulated a clustering loss function based on pairwise constraints, which regularized the joint probability distribution of latent variables and cluster labels. Comparative experiments with other deep clustering methods demonstrated the excellent performance of VDEPV.</p></abstract>

Nested Monte Carlo simulation of ionic systems with the primitive model using Debye-Hückel (DH) potential as an importance function and optimizing the DH potential with Kullback-Leibler divergence minimization

Nested Monte Carlo simulation of ionic systems with the primitive model using Debye-Hückel (DH) potential as an importance function and optimizing the DH potential with Kullback-Leibler divergence minimization

Multisource Fusion UAV Cluster Cooperative Positioning Using Information Geometry

Due to the functional limitations of a single UAV, UAV clusters have become an important part of smart cities, and the relative positioning between UAVs is the core difficulty in UAV cluster applications. Existing UAVs can be equipped with satellite navigation, radio navigation, and other positioning equipment, but in complex environments, such as urban canyons, various navigation sources cannot achieve full positioning information due to occlusion, interference, and other factors, and existing positioning fusion methods cannot meet the requirements of these environments. Therefore, demand exists for the real-time positioning of UAV clusters. Aiming to solve the above problems, this paper proposes multisource fusion UAV cluster cooperative positioning using information geometry (UCP-IG), which converts various types of navigation source information into information geometric probability models and reduces the impact of accidental errors, and proposes the Kullback–Leibler divergence minimization (KLM) fusion method to achieve rapid fusion on geometric manifolds and creatively solve the problem of difficult fusion caused by different positioning information formats and parameters. The method proposed in this paper is compared with the main synergistic methods, such as LS and neural networks, in an ideal scenario, a mutation error scenario, and a random motion scenario. The simulation results show that by using UAV cluster movement, the method proposed in this paper can effectively suppress mutation errors and achieve fast positioning.

Poisson Multi-Bernoulli Approximations for Multiple Extended Object Filtering

The Poisson multi-Bernoulli mixture (PMBM) is a multiobject conjugate prior for the closed-form Bayes random finite set filter. The extended object PMBM filter provides a closed-form solution for multiple extended object filtering with standard models. This article considers computationally lighter alternatives to the extended object PMBM filter by propagating a Poisson multi-Bernoulli (PMB) density through the filtering recursion. A new local hypothesis representation is presented, where each measurement creates a new Bernoulli component. This facilitates the developments of methods for efficiently approximating the PMBM posterior density after the update step as a PMB. Based on the new hypothesis representation, two approximation methods are presented: one is based on the track-oriented multi-Bernoulli (MB) approximation, and the other is based on the variational MB approximation via Kullback–Leibler divergence minimization. The performance of the proposed PMB filters with gamma Gaussian inverse-Wishart implementations are evaluated in a simulation study.

An Iterative Nonlinear Filter Based on Posterior Distribution Approximation via Penalized Kullback–Leibler Divergence Minimization

This letter deals with Gaussian approximation of complicated posterior distribution involved in the Bayesian paradigm for nonlinear dynamic systems. A general formulation for Gaussian approximation is first provided by equivalently representing posterior distribution as a Gaussian one with some constraint via embedding technique. In this work, it is specified as a penalized Kullback–Leibler divergence minimization problem. This minimization is solved for the expected Gaussian approximation by utilizing a pre-selected cubature rule and the conditional gradient method. Then, a novel iterative filter is developed for nonlinear dynamic systems. In addition, it is also proved to be optimal in linear cases and demonstrated to be effective through simulations.

Variational Bayesian Kalman filter using natural gradient

We propose a technique based on the natural gradient method for variational lower bound maximization for a variational Bayesian Kalman filter. The natural gradient approach is applied to the Kullback-Leibler divergence between the parameterized variational distribution and the posterior density of interest. Using a Gaussian assumption for the parametrized variational distribution, we obtain a closed-form iterative procedure for the Kullback-Leibler divergence minimization, producing estimates of the variational hyper-parameters of state estimation and the associated error covariance. Simulation results in both a Doppler radar tracking scenario and a bearing-only tracking scenario are presented, showing that the proposed natural gradient method outperforms existing methods which are based on other linearization techniques in terms of tracking accuracy.

A Poisson multi-Bernoulli mixture filter with spawning based on Kullback–Leibler divergence minimization

In the classical form, the Poisson Multi-Bernoulli Mixture (PMBM) filter uses a PMBM density to describe target birth, surviving, and death, which does not model the appearance of spawned targets. Although such a model can handle target birth, surviving, and death well, its performance may degrade when target spawning arises. The reason for this is that the original PMBM filter treats the spawned targets as birth targets, ignoring the surviving targets’ information. In this paper, we propose a Kullback–Leibler Divergence (KLD) minimization based derivation for the PMBM prediction step, including target spawning, in which the spawned targets are modeled using a Poisson Point Process (PPP). Furthermore, to improve the computational efficiency, three approximations are used to implement the proposed algorithm, such as the Variational Multi-Bernoulli (VMB) filter, the Measurement-Oriented marginal MeMBer/Poisson (MOMB/P) filter, and the Track-Oriented marginal MeMBer/Poisson (TOMB/P) filter. Finally, simulation results demonstrate the validity of the proposed filter by using the spawning model in these three approximations.

Effective Sea Clutter Suppression via MIMO Radar Space–Time Adaptive Processing Strategy

Due to nonlinear varying motions of the physical sea surface, sea clutter suppression is challenging with the broadened and rapidly changed Doppler spectrum content. In this article, an effective sea clutter suppression scheme based on multiple-input–multiple-output (MIMO) radar is presented via studying sea clutter features and reconstructing clutter spatial-Doppler spectrum accurately. To depict complicated Doppler features more precisely, the multistage Gaussian mixture model (GMM) is first developed in this article. In terms of exploring the dominant Doppler characteristics, GMM is more applicable in dynamic sea environments, including main swell and breaking waves. Afterward, for accurate reconstruction of clutter spectrum, the MIMO radar Gaussian mixture prior space–time adaptive processing (GMP-STAP) scheme is proposed for accurate estimation of clutter features. The Kullback–Leibler (KL) divergence is developed, and approximate distribution of posterior probability density functions (PDF) can be explored with low information loss based on complicated prior structure. Closed-form solutions of variables and parameters can be formulated through the minimization of KL divergence. The robust multistage Doppler power spectrum of sea clutter can be reconstructed accordingly. Moreover, considering that the independent and identically distributed (IID) property is easily broken among reference samples, the effective cross-correlation analysis is launched to keep statistical characteristics identical and reduce statistical errors. Finally, according to the experiments, accurate reconstruction of sea clutter features can be realized, and effective sea clutter suppression performance can be illustrated.

Robust source camera identification against adversarial attacks

Application of Deep Neural Networks (DNN) has dramatically improved the performance of Source Camera Identification (SCI), but easily suffers from adversarial attacks. These attacks raise security problems by tampering the identified outcomes with imperceptible noise. To address this issue, we analyze the feature extraction mapping for DNN-based SCI models on manifolds and discover that the vulnerability comes from the oscillation of the mapping. In light of this, we take that the feature extraction mapping should satisfy locally smooth and information monotonicity as a new design principle for robust SCI, and accordingly developed a defensive scheme. The proposed scheme constructs local smooth mapping that guarantees information monotonicity and achieves sufficient statistics by minimizing Kullback Leibler Divergence (KLD) between the local statistic coordinates on two manifolds. To enhance the usability of our method, we implement it with a Pre-Defense Network (PDN) trained by a two-phase training strategy, which ensures robustness, accuracy, and portability. Experiments on Dresden Image Dataset demonstrate that the proposed defense method offers not only strong robustness for the DNN-based SCI model against adversarial attacks, but also yields comparable or even superior identification performance over existing defense methods. Moreover, PDN also shows defense effect when migrated to other DNN-based SCI models, without extra retraining.

Convergence and stability of a micro–macro acceleration method: Linear slow–fast stochastic differential equations with additive noise

We analyze the convergence and stability of a micro–macro acceleration algorithm for Monte Carlo simulations of linear stiff stochastic differential equations with a time-scale separation between the fast evolution of the individual stochastic realizations and some slow macroscopic state variables of the process. The micro–macro acceleration method performs a short simulation of a large ensemble of individual fast paths, before extrapolating the macroscopic state variables of interest over a larger time step. After extrapolation, the method constructs a new probability distribution that is consistent with the extrapolated macroscopic state variables, while minimizing Kullback–Leibler divergence with respect to the distribution available at the end of the Monte Carlo simulation. In the current work, we study the convergence and stability of this method on linear stochastic differential equations with additive noise, when only extrapolating the mean of the slow component. For this case, we prove convergence to the microscopic dynamics when the initial distribution is Gaussian and present a stability result for non-Gaussian initial laws.

Physics-Prior Bayesian Neural Networks in Semiconductor Processing

With the fast scaling-down and evolution of integrated circuit (IC) manufacturing technology, the fabrication process becomes highly complex, and the experimental cost of the processes is significantly elevated. Therefore, in many cases, it is very costly to obtain a sufficient amount of experimental data. To develop an efficient method to predict the results of semiconductor experiments with a small amount of known data, we use a novel method based on Bayesian framework with the prior distribution constructed by technology computer-aided-design (TCAD) physical models. This method combines the advantages of statistical models and physical models in the aspect that TCAD can provide visionary guidance on an experiment when a limited amount of experimental data is available, and a machine learning model can account for subtle anomalous effects. Specifically, we use aspect ratio dependent etching (ARDE) phenomenon as an example and use variational inference with Kullback-Leibler divergence minimization to achieve the approximation to the posterior distribution. The relation between etching process input parameters and etching depth is learned using the Bayesian neural network with TCAD priors. Using this method with 35 neurons per hidden layer, mean square error (MSE) in the test set is reduced from 0.2896 to 0.0175, 0.058 to 0.0183, 0.0563 to 0.0188, 0.058 to 0.019 for partition=10, 20, 30, 40, respectively, reference to the baseline BNN where a regular normal distribution prior with zero mean and unity standard deviation N(0,1) is used.

Recursive Maximum Correntropy Learning Algorithm With Adaptive Kernel Size

In this brief, a robust correntropy-based adaptive learning algorithm, called the adaptive kernel recursive maximum correntropy criterion, is proposed by considering an adaptive kernel size based on the Kullback–Leibler divergence minimization. Simulation results confirm that the proposed algorithm can perform better than other adaptive algorithms especially when the environment is disturbed by non-Gaussian noises. The proposed algorithm is beneficial for using in helicopters, since the pilot speech signal is contaminated by the acoustic impulsive noises that are originated from rotor blades.

L1-Minimization Algorithm for Bayesian Online Compressed Sensing

In this work, we propose a Bayesian online reconstruction algorithm for sparse signals based on Compressed Sensing and inspired by L1-regularization schemes. A previous work has introduced a mean-field approximation for the Bayesian online algorithm and has shown that it is possible to saturate the offline performance in the presence of Gaussian measurement noise when the signal generating distribution is known. Here, we build on these results and show that reconstruction is possible even if prior knowledge about the generation of the signal is limited, by introduction of a Laplace prior and of an extra Kullback–Leibler divergence minimization step for hyper-parameter learning.

Derivation of the PHD and CPHD Filters Based on Direct Kullback–Leibler Divergence Minimization

In this paper, we provide novel derivations of the probability hypothesis density (PHD) and cardinalised PHD (CPHD) filters without using probability generating functionals or functional derivatives. We show that both the PHD and CPHD filters fit in the context of assumed density filtering and implicitly perform Kullback–Leibler divergence (KLD) minimizations after the prediction and update steps. We perform the KLD minimizations directly on the multitarget prediction and posterior densities.

Posterior Linearization Filter: Principles and Implementation Using Sigma Points

This paper is concerned with Gaussian approximations to the posterior probability density function (PDF) in the update step of Bayesian filtering with nonlinear measurements. In this setting, sigma-point approximations to the Kalman filter (KF) recursion are widely used due to their ease of implementation and relatively good performance. In the update step, these sigma-point KFs are equivalent to linearizing the nonlinear measurement function by statistical linear regression (SLR) with respect to the prior PDF. In this paper, we argue that the measurement function should be linearized using SLR with respect to the posterior rather than the prior to take into account the information provided by the measurement. The resulting filter is referred to as the posterior linearization filter (PLF). In practice, the exact PLF update is intractable but can be approximated by the iterated PLF (IPLF), which carries out iterated SLRs with respect to the best available approximation to the posterior. The IPLF can be seen as an approximate recursive Kullback-Leibler divergence minimization procedure. We demonstrate the high performance of the IPLF in relation to other Gaussian filters in two numerical examples.

A reweighted $l^2$ method for image restoration with Poisson and mixed Poisson-Gaussian noise

We study weighted $l^2$ fidelity in variational models for Poisson noise related image restoration problems. Gaussian approximation to Poisson noise statistic is adopted to deduce weighted $l^2$ fidelity. Different from the traditional weighted $l^2$ approximation, we propose a reweighted $l^2$ fidelity with sparse regularization by wavelet frame. Based on the split Bregman algorithm introduced in [21], the proposed numerical scheme is composed of three easy subproblems that involve quadratic minimization, soft shrinkage and matrix vector multiplications. Unlike usual least square approximation of Poisson noise, we dynamically update the underlying noise variance from previous estimate. The solution of the proposed algorithm is shown to be the same as the one obtained by minimizing Kullback-Leibler divergence fidelity with the same regularization. This reweighted $l^2$ formulation can be easily extended to mixed Poisson-Gaussian noise case. Finally, the efficiency and quality of the proposed algorithm compared to other Poisson noise removal methods are demonstrated through denoising and deblurring examples. Moreover, mixed Poisson-Gaussian noise tests are performed on both simulated and real digital images for further illustration of the performance of the proposed method.

Vector Quantization by Minimizing Kullback-Leibler Divergence between the Class Label Distributions over Quantization Input and Output

This paper proposes a new method for vector quantization by minimizing the Divergence of Kullback-Leibler between the class label distributions over the quantization inputs, which are original vectors, and the output, which is the quantization subsets of the vector set. In this way, the vector quantization output can keep as much information of the class label as possible. An objective function is constructed and we developed an iterative algorithm to minimize it as well. The novel method is evaluated on bag-of-features based image classification problems.

Optimizing information using the EM algorithm in item response theory

Latent trait models such as item response theory (IRT) hypothesize a functional relationship between an unobservable, or latent, variable and an observable outcome vari- able. In educational measurement, a discrete item response is usually the observable out- come variable, and the latent variable is associated with an examinee's trait level (e.g., skill, proficiency). The link between the two variables is called an item response function. This function, defined by a set of item parameters, models the probability of observing a given item response, conditional on a specific trait level. Typically in a measurement setting, neither the item parameters nor the trait levels are known, and so must be estimated from the pattern of observed item responses. Although a maximum likelihood approach can be taken in estimating these parameters, it usually cannot be employed directly. Instead, a method of marginal maximum likelihood (MML) is utilized, via the expectation-maximization (EM) algorithm. Alternating between an expectation (E) step and a maximization (M) step, the EM algorithm assures that the marginal log likelihood function will not decrease after each EM cycle, and will converge to a local maximum. Interestingly, the negative of this marginal log likelihood function is equal to the relative entropy, or Kullback-Leibler divergence, between the conditional distribution of the latent variables given the observable variables and the joint likelihood of the latent and observable variables. With an unconstrained optimization for the M-step proposed here, the EM algo- rithm as minimization of Kullback-Leibler divergence admits the convergence results due to Csiszar and Tusnady (Statistics & Decisions, 1:205-237, 1984), a consequence of the bi- nomial likelihood common to latent trait models with dichotomous response variables. For this unconstrained optimization, the EM algorithm converges to a global maximum of the marginal log likelihood function, yielding an information bound that permits a fixed point of reference against which models may be tested. A likelihood ratio test between marginal log likelihood functions obtained through constrained and unconstrained M-steps is pro- vided as a means for testing models against this bound. Empirical examples demonstrate the

A Stochastic Optimization Approach for Joint Relay Assignment and Power Allocation in Orthogonal Amplify-and-Forward Cooperative Wireless Networks

This paper addresses the joint relay assignment and power allocation problem for orthogonal multiuser systems using amplify-and-forward (AF) relaying nodes in the downlink. Our aim is to maximize the sum-rate subject to individual and total power constraints on the relays and a relay assignment constraint. In the case of fixed relay selection, the power allocation optimization is convex and an efficient recursive algorithm is proposed to achieve the optimum. The joint optimization of relay selection and power allocation, however, appears to be non-convex and is not known to be tractable. To tackle this, we propose a novel algorithm using Markov chain Monte-Carlo with Kullback-Leibler divergence minimization (MCMC-KLDM), which is proved to converge to the global optimum almost surely. Results show that the proposed scheme significantly outperforms a greedy approach and achieves near-optimal performance at very low complexity.