Divergence Minimization Research Articles

CLSESSP: Contrastive learning of sentence embedding with strong semantic prototypes

By drawing inspiration from the computer vision domain, contrastive learning has emerged as another paradigm for unsupervised sentence embedding. By using contrastive learning, most recent sentence embedding methods have achieved promising results. However, previous methods focused on constructing positive and negative pairs through traditional data augmentation, which was unable to balance the loss of semantics and the additional features brought by aggressive data augmentation. In the present study, we propose a prototype contrastive learning framework based on a distribution divergence minimization loss function and introduce the concept of strong semantic prototypes. Specifically, with the aid of prompts, our method constructs three semantic prototypes for each instance: a basic semantic prototype, a strong semantic prototype and a negative prototype. By combining InfoNCE loss with distribution divergence minimization loss, Contrastive Learning of Sentence Embedding with Strong Semantic Prototypes (CLSESSP) creates another optimization objective that integrates strong augmentation. The experiments conducted on the semantic text similarity (STS) datasets demonstrate that the proposed CLSESSP surpasses the strong baseline by an average of 2.9 points in the BERT-base model. Extensive experimental results on transfer and clustering tasks demonstrate the effectiveness of our proposed model compared to strong baselines. The code is available at https://github.com/KCshen1125/CLSESSP.

Directional TV algorithm for fast EPR imaging

Precise radiation guided by oxygen images has demonstrated superiority over the traditional radiation methods. Electron paramagnetic resonance (EPR) imaging has proven to be the most advanced oxygen imaging modality. However, the main drawback of EPR imaging is the long scan time. For each projection, we usually need to collect the projection many times and then average them to achieve high signal-to-noise ratio (SNR). One approach to fast scan is to reduce the repeating time for each projection. While the projections would be noisy and thus the traditional commonly-use filtered backprojection (FBP) algorithm would not be capable of accurately reconstructing images. Optimization-based iterative algorithms may accurately reconstruct images from noisy projections for they may incorporate prior information into optimization models. Based on the total variation (TV) algorithms for EPR imaging, in this work, we propose a directional TV (DTV) algorithm to further improve the reconstruction accuracy. We construct the DTV constrained, data divergence minimization (DTVcDM) model, derive its Chambolle–Pock (CP) solving algorithm, validate the correctness of the whole algorithm, and perform evaluations via simulated and real data. The experimental results show that the DTV algorithm outperforms the existing TV and FBP algorithms in fast EPR imaging. Compared to the standard FBP algorithm, the proposed algorithm may achieve 10 times of acceleration.

Cumulant GAN.

In this article, we propose a novel loss function for training generative adversarial networks (GANs) aiming toward deeper theoretical understanding as well as improved stability and performance for the underlying optimization problem. The new loss function is based on cumulant generating functions (CGFs) giving rise to Cumulant GAN. Relying on a recently derived variational formula, we show that the corresponding optimization problem is equivalent to Rényi divergence minimization, thus offering a (partially) unified perspective of GAN losses: the Rényi family encompasses Kullback-Leibler divergence (KLD), reverse KLD, Hellinger distance, and χ2 -divergence. Wasserstein GAN is also a member of cumulant GAN. In terms of stability, we rigorously prove the linear convergence of cumulant GAN to the Nash equilibrium for a linear discriminator, Gaussian distributions, and the standard gradient descent ascent algorithm. Finally, we experimentally demonstrate that image generation is more robust relative to Wasserstein GAN and it is substantially improved in terms of both inception score (IS) and Fréchet inception distance (FID) when both weaker and stronger discriminators are considered.

Robust Clustering with Normal Mixture Models: A Pseudo [formula omitted]-Likelihood Approach

As in other estimation scenarios, likelihood based estimation in the normal mixture set-up is highly non-robust against model misspecification and presence of outliers (apart from being an ill-posed optimization problem). A robust alternative to the ordinary likelihood approach for this estimation problem is proposed which performs simultaneous estimation and data clustering and leads to subsequent anomaly detection. To invoke robustness, the methodology based on the minimization of the density power divergence (or alternatively, the maximization of the β-likelihood) is utilized under suitable constraints. An iteratively reweighted least squares approach has been followed in order to compute the proposed estimators for the component means (or equivalently cluster centers) and component dispersion matrices which leads to simultaneous data clustering. Some exploratory techniques are also suggested for anomaly detection, a problem of great importance in the domain of statistics and machine learning. The proposed method is validated with simulation studies under different set-ups; it performs competitively or better compared to the popular existing methods like K-medoids, TCLUST, trimmed K-means and MCLUST, especially when the mixture components (i.e., the clusters) share regions with significant overlap or outlying clusters exist with small but non-negligible weights (particularly in higher dimensions). Two real datasets are also used to illustrate the performance of the newly proposed method in comparison with others along with an application in image processing. The proposed method detects the clusters with lower misclassification rates and successfully points out the outlying (anomalous) observations from these datasets.

Deterministic, quenched, and annealed parameter estimation for heterogeneous network models.

At least two different approaches to define and solve statistical models for the analysis of economic systems exist: the typical, econometric one, interpreting the gravity model specification as the expected link weight of an arbitrary probability distribution, and the one rooted in statistical physics, constructing maximum-entropy distributions constrained to satisfy certain network properties. In a couple of recent companion papers, they have been successfully integrated within the framework induced by the constrained minimization of the Kullback-Leibler divergence: specifically, two broad classes of models have been devised, i.e., the integrated and conditional ones, defined by different, probabilistic rules to place links, load them with weights and turn them into proper, econometric prescriptions. Still, the recipes adopted by the two approaches to estimate the parameters entering into the definition of each model differ. In econometrics, a likelihood that decouples the binary and weighted parts of a model, treating a network as deterministic, is typically maximized; to restore its random character, two alternatives exist: either solving the likelihood maximization on each configuration of the ensemble and taking the average of the parameters afterwards or taking the average of the likelihood function and maximizing the latter one. The difference between these approaches lies in the order in which the operations of averaging and maximization are taken-a difference that is reminiscent of the quenched and annealed ways of averaging out the disorder in spin glasses. The results of the present contribution, devoted to comparing these recipes in the case of continuous, conditional network models, indicate that the annealed estimation recipe represents the best alternative to the deterministic one.

Data-Driven Clustering and Bernoulli Merging for the Poisson Multi-Bernoulli Mixture Filter

This paper proposes a clustering and merging approach for the Poisson multi-Bernoulli mixture (PMBM) filter to lower its computational complexity and make it suitable for multiple target tracking with a high number of targets. We define a measurement-driven clustering algorithm to reduce the data association problem into several subproblems, and we provide the derivation of the resulting clustered PMBM posterior density via Kullback-Leibler divergence minimisation. Furthermore, we investigate different strategies to reduce the number of single target hypotheses by approximating the posterior via merging and inter-track swapping of Bernoulli components. We evaluate the performance of the proposed algorithm on simulated tracking scenarios with more than one thousand targets.

The Multiple Model Poisson Multi-Bernoulli Mixture Filter for Extended Target Tracking

In an intelligent transportation system (ITS), the single motion model is not enough to meet the needs of simulation systems (or practical applications) and guarantee good tracking performance for multiple maneuvering extended target tracking (ETT). This paper presents multiple models Poisson multi-Bernoulli mixture (MM-PMBM) filter for extended target tracking to address this problem. To make multiple models transform in an accurate manner, the jump Markov system (JMS) is timely introduced in filtering recursion. After that, the model probability of target is recursively updated and used to estimate the target position. In addition, we derive the closed-form solution to the proposed filter based on the Gamma Gaussian inverse-Wishart (GGIW) implementation. The approach to handling the merging problem of GGIW components via Kullback-Leibler divergence (KLD) minimization is also presented, resulting in better utilization of available components. Simulation results demonstrate that the proposed filter has superior performance in comparison to several state-of-the-art multi-target filters.

Nested Monte Carlo simulation of ionic systems with the primitive model using Debye-Hückel (DH) potential as an importance function and optimizing the DH potential with Kullback-Leibler divergence minimization

Nested Monte Carlo simulation of ionic systems with the primitive model using Debye-Hückel (DH) potential as an importance function and optimizing the DH potential with Kullback-Leibler divergence minimization

Managing Conflicting Interests of Stakeholders in Influencer Marketing

A successful campaign should be able to attract investment from the brand, and meanwhile manage the conflicting interests in the campaign cost between the brand and the influencers. As such, the agency between these two stakeholders plays a vital role. Motivated by the above, we stand in the agency's shoes to formulate an interesting yet practical problem, namely Profit Divergence Minimization in Investment-Persuasive Influencer Marketing Campaign (PDMIC). This problem aims to (i) minimize the divergence of the actual hiring prices from the asking prices of the influencers and meanwhile (ii) maintain the attractiveness of the pricing scheme for the influencers to the brand. We show that this problem is NP-hard. To mitigate the challenge of the extremely large searching space of the hiring prices of the influencers, we solve this problem by firstly considering a restrictive searching sub-space and then gradually expanding the searching sub-space to the whole space in the end (specifically, from binary price choices to a set of integer prices and then to any price in the feasible price range). We propose effective yet efficient approximate algorithms for solving the problem in each of these settings. Extensive experiments demonstrate the superiority of our methods.

A hybrid method for density power divergence minimization with application to robust univariate location and scale estimation

We develop a new globally convergent optimization method for solving a constrained minimization problem underlying the minimum density power divergence estimator for univariate Gaussian data in the presence of outliers. Our hybrid procedure combines classical Newton’s method with a gradient descent iteration equipped with a step control mechanism based on Armijo’s rule to ensure global convergence. Extensive simulations comparing the resulting estimation procedure with the more prominent robust competitor, Minimum Covariance Determinant (MCD) estimator, across a wide range of breakdown point values suggest improved efficiency of our method. Application to estimation and inference for a real-world dataset is also given.

Sampling by divergence minimization

We introduce a Markov Chain Monte Carlo (MCMC) method that is designed to sample from target distributions with irregular geometry using an adaptive scheme. In cases where targets exhibit non-Gaussian behavior, we propose that adaptation should be regional rather than global. Our algorithm minimizes the information projection component of the Kullback–Leibler (KL) divergence between the proposal and target distributions to encourage proposals that are distributed similarly to the regional geometry of the target. Unlike traditional adaptive MCMC, this procedure rapidly adapts to the geometry of the target’s current position as it explores the surrounding space without the need for many preexisting samples. The divergence minimization algorithms are tested on target distributions with irregularly shaped modes and we provide results demonstrating the effectiveness of our methods.

Subdomain Adaptation With Manifolds Discrepancy Alignment.

Reducing domain divergence is a key step in transfer learning. Existing works focus on the minimization of global domain divergence. However, two domains may consist of several shared subdomains, and differ from each other in each subdomain. In this article, we take the local divergence of subdomains into account in transfer. Specifically, we propose to use the low-dimensional manifold to represent the subdomain, and align the local data distribution discrepancy in each manifold across domains. A manifold maximum mean discrepancy (M3D) is developed to measure the local distribution discrepancy in each manifold. We then propose a general framework, called transfer with manifolds discrepancy alignment (TMDA), to couple the discovery of data manifolds with the minimization of M3D. We instantiate TMDA in the subspace learning case considering both the linear and nonlinear mappings. We also instantiate TMDA in the deep learning framework. Experimental studies show that TMDA is a promising method for various transfer learning tasks.

Multisource Fusion UAV Cluster Cooperative Positioning Using Information Geometry

Due to the functional limitations of a single UAV, UAV clusters have become an important part of smart cities, and the relative positioning between UAVs is the core difficulty in UAV cluster applications. Existing UAVs can be equipped with satellite navigation, radio navigation, and other positioning equipment, but in complex environments, such as urban canyons, various navigation sources cannot achieve full positioning information due to occlusion, interference, and other factors, and existing positioning fusion methods cannot meet the requirements of these environments. Therefore, demand exists for the real-time positioning of UAV clusters. Aiming to solve the above problems, this paper proposes multisource fusion UAV cluster cooperative positioning using information geometry (UCP-IG), which converts various types of navigation source information into information geometric probability models and reduces the impact of accidental errors, and proposes the Kullback–Leibler divergence minimization (KLM) fusion method to achieve rapid fusion on geometric manifolds and creatively solve the problem of difficult fusion caused by different positioning information formats and parameters. The method proposed in this paper is compared with the main synergistic methods, such as LS and neural networks, in an ideal scenario, a mutation error scenario, and a random motion scenario. The simulation results show that by using UAV cluster movement, the method proposed in this paper can effectively suppress mutation errors and achieve fast positioning.

An iterative reconstruction algorithm without system matrix for EPR imaging

Electron paramagnetic resonance (EPR) imaging is an advanced oxygen imaging modality for oxygen-image guided radiation. The iterative reconstruction algorithm is the research hot-point in image reconstruction for EPR imaging (EPRI) for this type of algorithm may incorporate image-prior information to construct advanced optimization model to achieve accurate reconstruction from sparse-view projections and/or noisy projections. However, the system matrix in the iterative algorithm needs complicated calculation and needs huge memory-space if it is stored in memory. In this work, we propose an iterative reconstruction algorithm without system matrix for EPRI to simplify the whole iterative reconstruction process. The function of the system matrix is to calculate the projections, whereas the function of the transpose of the system matrix is to perform backprojection. The existing projection and backprojection methods are all based on the configuration that the imaged-object remains stationary and the scanning device rotates. Here, we implement the projection and backprojection operations by fixing the scanning device and rotating the object. Thus, the core algorithm is only the commonly-used image-rotation algorithm, while the calculation and store of the system matrix are avoided. Based on the idea of image rotation, we design a specific iterative reconstruction algorithm for EPRI, total variation constrained data divergence minimization (TVcDM) algorithm without system matrix, and named it as image-rotation based TVcDM (R-TVcDM). Through a series of comparisons with the original TVcDM via real projection data, we find that the proposed algorithm may achieve similar reconstruction accuracy with the original one. But it avoids the complicated calculation and store of the system matrix. The insights gained in this work may be also applied to other imaging modalities, for example computed tomography and positron emission tomography.

Alpha-divergence minimization for deep Gaussian processes

This paper proposes the minimization of α-divergences for approximate inference in the context of deep Gaussian processes (DGPs). The proposed method can be considered as a generalization of variational inference (VI) and expectation propagation (EP), two previously used methods for approximate inference in DGPs. Both VI and EP are based on the minimization of the Kullback-Leibler divergence. The proposed method is based on a scalable version of power expectation propagation, a method that introduces an extra parameter α that specifies the targeted α-divergence to be optimized. In particular, such a method can recover the VI solution when α→0 and the EP solution when α→1. An exhaustive experimental evaluation shows that the minimization of α-divergences via the proposed method is feasible in DGPs and that choosing intermediate values of the α parameter between 0 and 1 can give better results in some problems. This means that one can improve the results of VI and EP when training DGPs. Importantly, the proposed method allows for stochastic optimization techniques, making it able to address datasets with several millions of instances.

Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition

In recent years, self-supervised representation learning for skeleton-based action recognition has been developed with the advance of contrastive learning methods. The existing contrastive learning methods use normal augmentations to construct similar positive samples, which limits the ability to explore novel movement patterns. In this paper, to make better use of the movement patterns introduced by extreme augmentations, a Contrastive Learning framework utilizing Abundant Information Mining for self-supervised action Representation (AimCLR) is proposed. First, the extreme augmentations and the Energy-based Attention-guided Drop Module (EADM) are proposed to obtain diverse positive samples, which bring novel movement patterns to improve the universality of the learned representations. Second, since directly using extreme augmentations may not be able to boost the performance due to the drastic changes in original identity, the Dual Distributional Divergence Minimization Loss (D3M Loss) is proposed to minimize the distribution divergence in a more gentle way. Third, the Nearest Neighbors Mining (NNM) is proposed to further expand positive samples to make the abundant information mining process more reasonable. Exhaustive experiments on NTU RGB+D 60, PKU-MMD, NTU RGB+D 120 datasets have verified that our AimCLR can significantly perform favorably against state-of-the-art methods under a variety of evaluation protocols with observed higher quality action representations. Our code is available at https://github.com/Levigty/AimCLR.

A robust Poisson multi-Bernoulli filter for multi-target tracking based on arithmetic average fusion

The coalescence and missed detection are two key challenges in Multi-Target Tracking (MTT). To balance the tracking accuracy and real-time performance, the existing Random Finite Set (RFS) based filters are generally difficult to handle the above problems simultaneously, such as the Track-Oriented marginal Multi-Bernoulli/Poisson (TOMB/P) and Measurement-Oriented marginal Multi-Bernoulli/Poisson (MOMB/P) filters. Based on the Arithmetic Average (AA) fusion rule, this paper proposes a novel fusion framework for the Poisson Multi-Bernoulli (PMB) filter, which integrates both the advantages of the TOMB/P filter in dealing with missed detection and the advantages of the MOMB/P filter in dealing with coalescence. In order to fuse the different PMB distributions, the Bernoulli components in different Multi-Bernoulli (MB) distributions are associated with each other by Kullback-Leibler Divergence (KLD) minimization. Moreover, an adaptive AA fusion rule is designed on the basis of the exponential fusion weights, which utilizes the TOMB/P and MOMB/P updates to solve these difficulties in MTT. Finally, by comparing with the TOMB/P and MOMB/P filters, the performance of the proposed filter in terms of accuracy and efficiency is demonstrated in three challenging scenarios.

Poisson Multi-Bernoulli Approximations for Multiple Extended Object Filtering

The Poisson multi-Bernoulli mixture (PMBM) is a multiobject conjugate prior for the closed-form Bayes random finite set filter. The extended object PMBM filter provides a closed-form solution for multiple extended object filtering with standard models. This article considers computationally lighter alternatives to the extended object PMBM filter by propagating a Poisson multi-Bernoulli (PMB) density through the filtering recursion. A new local hypothesis representation is presented, where each measurement creates a new Bernoulli component. This facilitates the developments of methods for efficiently approximating the PMBM posterior density after the update step as a PMB. Based on the new hypothesis representation, two approximation methods are presented: one is based on the track-oriented multi-Bernoulli (MB) approximation, and the other is based on the variational MB approximation via Kullback–Leibler divergence minimization. The performance of the proposed PMB filters with gamma Gaussian inverse-Wishart implementations are evaluated in a simulation study.

Fault Knowledge Transfer Assisted Ensemble Method for Remaining Useful Life Prediction

Machinery remaining useful life (RUL) prediction is an important task in condition-based maintenance. Data-driven methods have been widely studied and applied, however, almost all the researches learn degradation trends regardless of different fault conditions, which can lead to different degradation patterns. This article proposes a novel fault information assisted RUL prediction method based on a convolutional long short-term memory (LSTM) ensemble network, where fault conditions are obtained via fault knowledge transfer. Divergence minimization and domain adversarial adaptation are combined to transfer fault knowledge from a fault dataset to the run-to-failure data in a weakly supervised manner. With the predicted fault information, the RUL prediction network can learn various degradation patterns under different faults separately using a structure of multiple LSTMs. Then an ensemble strategy based on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">soft fault conditions</i> is designed to get final RUL prediction results. Experiment on bearing datasets verifies the effectiveness of our proposed method.

Quantum field theories, Markov random fields and machine learning

The transition to Euclidean space and the discretization of quantum field theories on spatial or space-time lattices opens up the opportunity to investigate probabilistic machine learning within quantum field theory. Here, we will discuss how discretized Euclidean field theories, such as the ϕ 4 lattice field theory on a square lattice, are mathematically equivalent to Markov fields, a notable class of probabilistic graphical models with applications in a variety of research areas, including machine learning. The results are established based on the Hammersley-Clifford theorem. We will then derive neural networks from quantum field theories and discuss applications pertinent to the minimization of the Kullback-Leibler divergence for the probability distribution of the ϕ 4 machine learning algorithms and other probability distributions.