KL Divergence Research Articles

Sonar image segmentation using a multi-spatial information constraint fuzzy C-means clustering algorithm based on KL divergence

Sonar image segmentation using a multi-spatial information constraint fuzzy C-means clustering algorithm based on KL divergence

Trainability barriers and opportunities in quantum generative modeling

AbstractQuantum generative models provide inherently efficient sampling strategies and thus show promise for achieving an advantage using quantum hardware. In this work, we investigate the barriers to the trainability of quantum generative models posed by barren plateaus and exponential loss concentration. We explore the interplay between explicit and implicit models and losses, and show that using quantum generative models with explicit losses such as the KL divergence leads to a new flavor of barren plateaus. In contrast, the implicit Maximum Mean Discrepancy loss can be viewed as the expectation value of an observable that is either low-bodied and provably trainable, or global and untrainable depending on the choice of kernel. In parallel, we find that solely low-bodied implicit losses cannot in general distinguish high-order correlations in the target data, while some quantum loss estimation strategies can. We validate our findings by comparing different loss functions for modeling data from High-Energy-Physics.

Augmented ELBO regularization for enhanced clustering in variational autoencoders

With significant advances in deep neural networks, various new algorithms have emerged that effectively model latent structures within data, surpassing traditional clustering methods. Each data point is expected to belong to a single cluster in a typical clustering algorithm. However, clustering based on variational autoencoders (VAEs) represents the expectation of the overall clusters, denoted as c=1,…,K in the KL divergence term. Consequently, the latent embedding z can be learned to exist across multiple clusters with relatively balanced probabilities, rather than being strongly associated with a specific cluster. This study introduces an additional regularizer to encourage the latent embedding z to have a strong affiliation with specific clusters. We introduce optimization methods to maximize the ELBO that includes the newly added regularization term and explore methods to eliminate computationally challenging terms. The positive impact of this regularization on clustering accuracy was verified by examining the variance of the final cluster probabilities. Furthermore, an enhancement in the clustering performance was observed when regularization was introduced.

Deconstructing Deep Active Inference: A Contrarian Information Gatherer.

Active inference is a theory of perception, learning, and decision making that can be applied to neuroscience, robotics, psychology, and machine learning. Recently, intensive research has been taking place to scale up this framework using Monte Carlo tree search and deep learning. The goal of this activity is to solve more complicated tasks using deep active inference. First, we review the existing literature and then progressively build a deep active inference agent as follows: we (1) implement a variational autoencoder (VAE), (2) implement a deep hidden Markov model (HMM), and (3) implement a deep critical hidden Markov model (CHMM). For the CHMM, we implemented two versions, one minimizing expected free energy, CHMM[EFE] and one maximizing rewards, CHMM[reward]. Then we experimented with three different action selection strategies: the ε-greedy algorithm as well as softmax and best action selection. According to our experiments, the models able to solve the dSprites environment are the ones that maximize rewards. On further inspection, we found that the CHMM minimizing expected free energy almost always picks the same action, which makes it unable to solve the dSprites environment. In contrast, the CHMM maximizing reward keeps on selecting all the actions, enabling it to successfully solve the task. The only difference between those two CHMMs is the epistemic value, which aims to make the outputs of the transition and encoder networks as close as possible. Thus, the CHMM minimizing expected free energy repeatedly picks a single action and becomes an expert at predicting the future when selecting this action. This effectively makes the KL divergence between the output of the transition and encoder networks small. Additionally, when selecting the action down the average reward is zero, while for all the other actions, the expected reward will be negative. Therefore, if the CHMM has to stick to a single action to keep the KL divergence small, then the action down is the most rewarding. We also show in simulation that the epistemic value used in deep active inference can behave degenerately and in certain circumstances effectively lose, rather than gain, information. As the agent minimizing EFE is not able to explore its environment, the appropriate formulation of the epistemic value in deep active inference remains an open question.

BEATRICE: Bayesian fine-mapping from summary data using deep variational inference.

We introduce a novel framework BEATRICE to identify putative causal variants from GWAS statistics. Identifying causal variants is challenging due to their sparsity and high correlation in the nearby regions. To account for these challenges, we rely on a hierarchical Bayesian model that imposes a binary concrete prior on the set of causal variants. We derive a variational algorithm for this fine-mapping problem by minimizing the KL divergence between an approximate density and the posterior probability distribution of the causal configurations. Correspondingly, we use a deep neural network as an inference machine to estimate the parameters of our proposal distribution. Our stochastic optimization procedure allows us to sample from the space of causal configurations, which we use to compute the posterior inclusion probabilities and determine credible sets for each causal variant. We conduct a detailed simulation study to quantify the performance of our framework against two state-of-the-art baseline methods across different numbers of causal variants and noise paradigms, as defined by the relative genetic contributions of causal and noncausal variants. We demonstrate that BEATRICE achieves uniformly better coverage with comparable power and set sizes, and that the performance gain increases with the number of causal variants. We also show the efficacy BEATRICE in finding causal variants from the GWAS study of Alzheimer's disease. In comparison to the baselines, only BEATRICE can successfully find the APOE ϵ2 allele, a commonly associated variant of Alzheimer's. BEATRICE is available for download at https://github.com/sayangsep/Beatrice-Finemapping.

Detection and Mitigation of Label-Flipping Attacks in FL Systems With KL Divergence

Detection and Mitigation of Label-Flipping Attacks in FL Systems With KL Divergence

Diffusion model conditioning on gaussian mixture model and negative gaussian mixture gradient

Diffusion models (DMs) are a type of generative model that has had a significant impact on image synthesis and beyond.They can incorporate a wide variety of conditioning inputs—such as text or bounding boxes—to guide generation. In this work, we introduce a novel conditioning mechanism that applies Gaussian mixture models (GMMs) for feature conditioning, which helps steer the denoising process in DMs. Drawing on set theory, our comprehensive theoretical analysis reveals that the conditional latent distribution based on features differs markedly from that based on classes. Consequently, feature-based conditioning tends to generate fewer defects than class-based conditioning. Experiments are designed and carried out and the experimental results support our theoretical findings as well as effectiveness of proposed feature conditioning mechanism. Additionally, we propose a new gradient function named the Negative Gaussian Mixture Gradient (NGMG) and incorporate it into the training of diffusion models alongside an auxiliary classifier. We theoretically demonstrate that NGMG offers comparable advantages to the Wasserstein distance, serving as a more effective cost function when learning distributions supported by low-dimensional manifolds, especially in contrast to many likelihood-based cost functions, such as KL divergences.

Mix-DDPM: Enhancing Diffusion Models through Fitting Mixture Noise with Global Stochastic Offset

Denoising diffusion probabilistic models (DDPM) have shown impressive performance in various domains as a class of deep generative models. In this article, we introduce the mixture noise-based DDPM (Mix-DDPM), which considers the Markov diffusion posterior as a Gaussian mixture model. Specifically, Mix-DDPM randomly selects a Gaussian component and then adds the chosen Gaussian noise, which can be demonstrated as a more efficient way to perturb the signals into a simple known distribution. We further define the reverse probabilistic model as a parameterized Gaussian mixture kernel. Due to the intractability in calculating the KL divergence between Gaussian mixture models, we derive a variational bound to maximize the likelihood, offering a concise formulation for optimizing the denoising model and valuable insights for designing the sampling strategies. Our theoretical derivation highlights that Mix-DDPM need only shift image which requires the inclusion of a global stochastic offset in both the diffusion and reverse processes , which can be efficiently implemented with just several lines of code. The global stochastic offset effectively fits a Gaussian mixture distribution enhancing the degrees of freedom of the entire diffusion model. Furthermore, we present three streamlined sampling strategies that interface with diverse fast dedicated solvers for diffusion ordinary differential equations, boosting the efficacy of image representation in the sampling phase and alleviating the issue of slow generation speed, thereby enhancing both efficiency and accuracy. Extensive experiments on benchmark datasets demonstrate the effectiveness of Mix-DDPM and its superiority over the original DDPM.

Suppressed possibilistic fuzzy c-means clustering based on shadow sets for noisy data with imbalanced sizes

Possibilistic fuzzy c-means clustering (PFCM) algorithm generates fewer overlapping clustering centers than the possibilistic c-means clustering (PCM) algorithm and possesses better noise immunity than fuzzy c-means clustering (FCM) algorithm. However, with the increasing noise intensity and number of clusters, PFCM still faces the problem of getting partially overlapping clustering centers or mislocated centers in noise regions. Moreover, the sample-size imbalance increasingly intensifies the difficulty of positioning centers of small clusters. To solve the above problems, a suppressed possibilistic fuzzy c-means clustering algorithm based on shadow sets (S-SPFCM) is proposed by introducing the shadow set theory and the "suppressed competitive learning" strategy. Firstly, KL divergence is introduced in the objective function of PFCM to increase the anti-noise robustness of fuzzy memberships against long-range noise and outliers. Secondly, to reduce the number of overlapping centers caused by possibilistic memberships, the shadow set theory is introduced to divide each class adaptively into three regions (core, shadow, and exclusion regions) by an uncertainty balance method. The suppressed competitive learning method is extended by modifying the memberships of points within the three regions, thus artificially guiding the iterative track of clustering centers. Meanwhile, to further reduce the influence of imbalanced sizes, a scheme to reset mislocated centers in the core and shadow regions is also designed. In addition, to improve the segmentation performance of S-SPFCM for noisy images, a suppressed possibilistic fuzzy c-means clustering algorithm based on shadow sets and local information (SL-SPFCM) is also proposed. The SL-SPFCM first improves the Euclidean distance by a distance filtering scheme. Then SL-SPFCM introduces the local median membership of each pixel into the KL divergence of the objective function. Finally, experiments on synthetic datasets and color images which are characteristic of imbalanced sizes and noise injection demonstrate the proposed S-SPFCM and SL-SPFCM algorithms achieve smaller center deviations and higher clustering accuracy compared with several state-of-the-art clustering algorithms.

3D mineral prospectivity modeling using deep adaptation network transfer learning: A case study of the Xiadian gold deposit, Eastern China

Mineralization distribution is commonly heterogeneous in space due to the various geology and structure conditions or fluid flow and evoluation, causing the possible difference in mineralization distribution regularity between shallow and deep at a deposit. This presents important challenges for three-dimensional mineral prospectivity modeling (3D MPM) for deep zones with the scarcity data conditions. Transfer learning has shown promising generalization performance in tasks involving shifts in data distribution, reducing reliance on labeled samples and enhancing learning capability with limited data. In this study, we propose an approach of 3D MPM, namely DAN-CBAM, based on the deep adaptation network (DAN) augmented with the convolutional block attention module (CBAM). It theoretically can harmonize the distribution of ore-controlling features between shallow and deep zones of deposits while effectively extract critical high-dimensional features and spatial patterns. The Xiadian orogenic gold deposit was selected as a case study to validate the approach. The DAN integrates multi-layer, multi-kernel adaptation at the top layer of the CNN, resulting in improved alignment of marginal distributions across domains. Metrics for distribution similarity such as Wasserstein distance, were reduced by 0.250 and KL divergence decreased by 0.032. Additionally, the inclusion of the CBAM module led to a lower MK-MMD loss and a faster convergence rate. And the DAN-CBAM model achieves superior prediction accuracy (0.85) compared to traditional deep neural network (DNN) models (0.81). These highlight CBAM's effectiveness in enhancing the model's ability to capture spatial similarities in mineralization. Furthermore, the area under the curve (AUC) value of the DAN-CBAM model (0.869) significantly outperforms traditional machine learning methods, including CNN (0.786) and Random Forest (0.703) models, underscoring its superior predictive efficiency in 3D MPM for deep mineralization. Therefore, the proposed DAN-CBAM model is promising to be applied in the 3D MPM, in particular for the deposits with different mineralization distribution regularities in space.

BEATRICE: Bayesian Fine-mapping from Summary Data using Deep Variational Inference.

We introduce a novel framework BEATRICE to identify putative causal variants from GWAS statistics. Identifying causal variants is challenging due to their sparsity and high correlation in the nearby regions. To account for these challenges, we rely on a hierarchical Bayesian model that imposes a binary concrete prior on the set of causal variants. We derive a variational algorithm for this fine-mapping problem by minimizing the KL divergence between an approximate density and the posterior probability distribution of the causal configurations. Correspondingly, we use a deep neural network as an inference machine to estimate the parameters of our proposal distribution. Our stochastic optimization procedure allows us to simultaneously sample from the space of causal configurations. We use these samples to compute the posterior inclusion probabilities and determine credible sets for each causal variant. We conduct a detailed simulation study to quantify the performance of our framework against two state-of-the-art baseline methods across different numbers of causal variants and different noise paradigms, as defined by the relative genetic contributions of causal and non-causal variants. We demonstrate that BEATRICE achieves uniformly better coverage with comparable power and set sizes, and that the performance gain increases with the number of causal variants. We also show the efficacy BEATRICE in finding causal variants from the GWAS study of Alzheimer's disease. In comparison to the baselines, only BEATRICE can successfully find the APOE allele, a commonly associated variant of Alzheimer's. Thus, we show that BEATRICE is a valuable tool to identify causal variants from eQTL and GWAS summary statistics across complex diseases and traits.

Disturbances of thalamus and prefrontal cortex contribute to cognitive aging: A structure-function coupling analysis based on KL divergence

Normal aging is accompanied by changes in brain structure and function associated with cognitive decline. Structural and functional abnormalities, particularly the prefrontal cortex (PFC) and subcortical regions, contributed to cognitive aging. However, it remains unclear how the synchronized changes in structure and function of individual brain regions affect the cognition in aging. Using 3D T1-weighted structural data and movie watching functional magnetic resonance imaging data in a sample of 422 healthy individuals (ages from 18 to 87 years), we constructed regional structure–function coupling (SFC) of cortical and subcortical regions by quantifying the distribution similarity of gray matter volume (GMV) and amplitude of low-frequency fluctuation (ALFF). Further, we investigated age-related changes in SFC and its relationship with cognition. With aging, increased SFC localized in PFC, thalamus and caudate nucleus, decreased SFC in temporal cortex, lateral occipital cortex and putamen. Moreover, the SFC in the PFC was associated with executive function and thalamus was associated with the fluid intelligence, and partially mediated age-related cognitive decline. Collectively, our results highlight that tighter structure–function synchron of the PFC and thalamus might contribute to age-related cognitive decline, and provide insight into the substrate of the thalamo-prefrontal pathway with cognitive aging.

GAP: A group-based automatic pruning algorithm via convolution kernel fusion

In recent years, the deployment and operation of convolution neural networks on edge devices with limited computing capabilities have become increasingly challenging due to the large network structure and computational cost. Currently, the mainstream structured pruning algorithms mainly compress the network at the filter or layer level. However, these methods introduce too much human intervention with large granularities, which may lead to unpredictable performance after compression. In this paper, we propose a group-based automatic pruning algorithm(GAP) via kernel fusion to automatically search for the optimal pruning structure in a more fine-grained manner. Specifically, we first adopt a novel nonlinear dimensionality reduction clustering algorithm to divide the filters of each convolution layer into groups of equal size. Afterwards, we encode the mutual distribution similarity of the kernels within each group, and its KL divergence is employed as an importance indicator to determine the retained kernel groups through weighted fusion. Subsequently, we introduce an intelligent searching module that automatically explore and optimize the pruned structure of each layer. Finally, the pruned filters are permutated to form a dense group convolution and fine-tuned. Sufficient experiments show that, on two image classification datasets, for five advanced CNN models, our GAP algorithm outperforms most extant SOTA schemes, reduces artificial intervention, and enables efficient end-to-end training of compact models.

Fund transfer fraud detection: Analyzing irregular transactions and customer relationships with self-attention and graph neural networks

This paper presents a method for identifying fraudulent fund transfers using real bank data, analyzing customer information, transactional activities, and customer relationships. The preprocessing step transforms high-dimensional, irregular transaction time series into regular time series, then further compresses them in a latent space using a self-attention-based autoencoder. To address the scarcity of fraudulent data samples and mitigate training issues caused by data imbalance, various deep generative models, including the conditional variational autoencoder and Wasserstein generative adversarial network, are applied to generate additional fraudulent raw data and augment fraud samples in latent space. The reparameterization trick is integrated into the encoder–decoder structure to boost the model’s generative capabilities. Additionally, a Graph Neural Network (GNN) is used to model customer relationships. The proposed approach utilizes end-to-end learning, integrating the autoencoder’s reconstruction loss, KL divergence loss (when reparameterization trick is applied), and classification loss for fraud detection. To optimize computational resources, neighborhood sampling for GNN is combined with mini-batch training for the autoencoder, improving both training efficiency and model reliability. Comprehensive experiments demonstrate the effectiveness of the proposed fusion network, highlighting the importance of each component and preprocessing step. For example, the areas under the precision–recall curves for fraud detection show notable improvements in our model. For suspicious transactions identified by the bank’s rules, other models range from 0.66% to 22.15%, while our model reached 27%. For non-suspicious transactions, other models range from 2.53% to 22.00%, with our model achieving 22.90%. This model has potential for wider applications in anomaly detection, particularly in datasets with irregular time series and complex customer relationships.

Learning the feature distribution similarities for online time series anomaly detection

Identifying anomalies in multi-dimensional sequential data is crucial for ensuring optimal performance across various domains and in large-scale systems. Traditional contrastive methods utilize feature similarity between different features extracted from multidimensional raw inputs as an indicator of anomaly severity. However, the complex objective functions and meticulously designed modules of these methods often lead to efficiency issues and a lack of interpretability. Our study introduces a structural framework called SimDetector, which is a Local–Global Multi-Scale Similarity Contrast network. Specifically, the restructured and enhanced GRU module extracts more generalized local features, including long-term cyclical trends. The multi-scale sparse attention module efficiently extracts multi-scale global features with pattern information. Additionally, we modified the KL divergence to suit the characteristics of time series anomaly detection, proposing a symmetric absolute KL divergence that focuses more on overall distribution differences. The proposed method achieves results that surpass or approach the State-of-the-Art (SOTA) on multiple real-world datasets and synthetic datasets, while also significantly reducing Multiply-Accumulate Operations (MACs) and memory usage.

Intrinsic Rewards for Exploration Without Harm From Observational Noise: A Simulation Study Based on the Free Energy Principle.

In reinforcement learning (RL), artificial agents are trained to maximize numerical rewards by performing tasks. Exploration is essential in RL because agents must discover information before exploiting it. Two rewards encouraging efficient exploration are the entropy of action policy and curiosity for information gain. Entropy is well established in the literature, promoting randomized action selection. Curiosity is defined in a broad variety of ways in literature, promoting discovery of novel experiences. One example, prediction error curiosity, rewards agents for discovering observations they cannot accurately predict. However, such agents may be distracted by unpredictable observational noises known as curiosity traps. Based on the free energy principle (FEP), this letter proposes hidden state curiosity, which rewards agents by the KL divergence between the predictive prior and posterior probabilities of latent variables. We trained six types of agents to navigate mazes: baseline agents without rewards for entropy or curiosity and agents rewarded for entropy and/or either prediction error curiosity or hidden state curiosity. We find that entropy and curiosity result in efficient exploration, especially both employed together. Notably, agents with hidden state curiosity demonstrate resilience against curiosity traps, which hinder agents with prediction error curiosity. This suggests implementing the FEP that may enhance the robustness and generalization of RL models, potentially aligning the learning processes of artificial and biological agents.

Multi-modal incomplete label information three-way bidirectional decision-making: Applications of disease assessment

Disease assessment involves two stages of decision-making process. The first process is to infer the development trend of the disease through the existing judgment conditions, which is to make a positive estimation (prediction) decision. The second process is to update the judgment conditions for different disease states and optimize the clinical assessment evidence by fusing disease progression data, that is, to make a reverse evaluation (optimization) decision by reviewing historical diagnostic and treatment data to summarize the best judgment conditions for the disease characteristics. Therefore, we construct a three-way bidirectional decision method for multi-modal incomplete labeling information, considering the mixed data features with no labels and labels, and the bidirectional decision features of positive prediction and reverse optimization. Specifically, the method first introduces semi-supervised learning (SSL) into the rough sets, constructs a rough sets model based on cosine similarity relation, generates pseudo-labels for incomplete information, and explains the positive prediction decision process principle. Second, the gradient space is constructed on the predictive information system by the three-way decision-making principle, the loss function formed by the combination of KL divergence and cross entropy loss function is used to find the direction of the fastest decline in attribute information value, and the optimal decision rule and the automatic updating strategy of the threshold value are obtained. At the same time, the principle of the reverse optimization decision process is explained. Finally, multimodal information from pathological tissue images is used to evaluate clinical progress problem. A pathological tissue images dataset includes 2000 colorectal cancer and 300 kidney disease cases used for simulation experiments. The proposed three-way bidirectional decision-making model can be widely applied to actual decision-making scenarios for evaluating clinical progress of different diseases. Compared with previous studies, this method can realize objective calculation and automatic update of threshold and loss values and effectively solve complex prediction and optimization decision-making problems.

Quantifying Quantum Chaos through Microcanonical Distributions of Entanglement

A characteristic feature of “quantum chaotic” systems is that their eigenspectra and eigenstates display universal statistical properties described by random matrix theory (RMT). However, eigenstates of local systems also encode structure beyond RMT. To capture this feature, we introduce a framework that allows us to compare the properties of eigenstates in local systems with those of pure random states. In particular, our framework defines a notion of distance between quantum state ensembles that utilizes the Kullback-Leibler divergence to compare the microcanonical distribution of entanglement entropy (EE) of eigenstates with a reference RMT distribution generated by pure random states (with appropriate constraints). This notion gives rise to a quantitative metric for quantum chaos that not only accounts for averages of the distributions but also higher moments. The differences in moments are compared on a highly resolved scale set by the standard deviation of the RMT distribution, which is exponentially small in system size. As a result, the metric can distinguish between chaotic and integrable behaviors and, in addition, quantify and compare the of chaos (in terms of proximity to RMT behavior) between two systems that are assumed to be chaotic. We implement our framework in local, minimally structured, Floquet random circuits, as well as a canonical family of many-body Hamiltonians, the mixed-field Ising model (MFIM). Importantly, for Hamiltonian systems, we find that the reference random distribution must be appropriately constrained to incorporate the effect of energy conservation in order to describe the ensemble properties of midspectrum eigenstates. The metric captures deviations from RMT across all models and parameters, including those that have been previously identified as strongly chaotic, and for which other diagnostics of chaos such as level spacing statistics look strongly thermal. In Floquet circuits, the dominant source of deviations is the second moment of the distribution, and this persists for all system sizes. For the MFIM, we find significant variation of the KL divergence in parameter space. Notably, we find a small region where deviations from RMT are minimized, suggesting that “maximally chaotic” Hamiltonians may exist in fine-tuned pockets of parameter space. Published by the American Physical Society 2024

An uncertainty-aware domain adaptive semantic segmentation framework

Semantic segmentation is significant to realize the scene understanding of autonomous driving. Due to the lack of annotated real-world data, the technology of domain adaptation is applied so that the model is trained on the synthetic data and inferred on the real data. However, this domain gap leads to aleatoric and epistemic uncertainty. These uncertainties link to the potential safety issue of autonomous driving in normal weather and adverse weather. In this study, we explore the scientific problem that has received sparse attention previously. We postulate that the Dual Attention module can mitigate the uncertainty in the task of semantic segmentation and provide some empirical study to validate it. Furthermore, the utilization of Kullback-Leibler divergence (KL divergence) helps the estimation of aleatoric uncertainty and boosts the robustness of the segmentation model. Our empirical study on the diverse datasets of semantic segmentation demonstrates the effectiveness of our method in normal and adverse weather. Our code is available at: https://github.com/liubo629/Seg-Uncertainty-dual-attention.

IoT traffic classification and anomaly detection method based on deep autoencoders

This study investigates anomaly detection of IoT device traffic using Convolutional Neural Networks (CNN) and Variational Autoencoders (VAE) to enhance the detection capability of security threats in IoT environments. A series of hardware configurations, software environments, and hyperparameters were utilized to optimize the training and testing processes of the models. The CNN model demonstrates robust classification performance, achieving an accuracy rate of 95.85% on the test dataset, effectively distinguishing between different types of IoT device traffic. Meanwhile, the VAE model exhibits proficient anomaly detection capabilities by effectively capturing abnormal patterns in the data using reconstruction loss and KL divergence. The combined use of CNN and VAE models offers a comprehensive solution to cybersecurity challenges in IoT environments. Future research directions include exploring diverse IoT traffic data, practical deployment for validation, and further optimization of model structures and parameters to improve performance and applicability.