Symmetric Kullback-Leibler Divergence Research Articles

An automatic anomaly application detection system in mobile devices using FL-HTR-DBN and SKLD-SED K means algorithms

The proliferation of mobile technology has given rise to a multitude of applications, among them those designed with malicious intent, aimed at compromising the integrity of mobile devices (MDs). To combat this issue, this study introduces an innovative anomaly application detection system leveraging Federated Learning in conjunction with a Hyperbolic Tangent Radial-Deep Belief Network (FL-HTR-DBN). This system operates through two distinct phases: training and testing. During the training phase, the system first extracts log files and transforms them into a structured format, harnessing the power of the Hadoop System. Subsequently, these structured logs are converted into vector representations using the Updating Gate-BERT (UG-BERT) technique, thereby facilitating feature extraction. These features are then annotated utilizing the Symmetric Kullback Leibler Divergence squared Euclidean distance-based K Means (SKLD-SED K Means) algorithm. The FL-HTR-DBN model is subsequently trained using these labelled features. The detected anomalies are hashed and securely stored within an index tree, alongside their corresponding hashed Media Access Control (MAC) addresses. In the testing phase, log files are cross-referenced with the hashed index tree to identify potential anomalies. Notably, this novel approach outperforms many valuable outcomes in comparison with the existing approaches ConAnomaly, QLLog and LogCAD in terms of precision 97.5, recall 97.1, accuracy 95.9, F-measure 93.9, sensitivity 94.8 and specificity 95.9.

Image Quality Assessment: Measuring Perceptual Degradation via Distribution Measures in Deep Feature Spaces.

This study aims to develop advanced and training-free full-reference image quality assessment (FR-IQA) models based on deep neural networks. Specifically, we investigate measures that allow us to perceptually compare deep network features and reveal their underlying factors. We find that distribution measures enjoy advanced perceptual awareness and test the Wasserstein distance (WSD), Jensen-Shannon divergence (JSD), and symmetric Kullback-Leibler divergence (SKLD) measures when comparing deep features acquired from various pretrained deep networks, including the Visual Geometry Group (VGG) network, SqueezeNet, MobileNet, and EfficientNet. The proposed FR-IQA models exhibit superior alignment with subjective human evaluations across diverse image quality assessment (IQA) datasets without training, demonstrating the advanced perceptual relevance of distribution measures when comparing deep network features. Additionally, we explore the applicability of deep distribution measures in image super-resolution enhancement tasks, highlighting their potential for guiding perceptual enhancements. The code is available on website. (https://github.com/Buka-Xing/Deep-network-based-distribution-measures-for-full-reference-image-quality-assessment).

Sea clutter radar target detector based on autoregressive sieve bootstrap

The autoregressive sieve (ARS) bootstrap method can capture the correlation of time series and expand the original data set to the required volume. In this study, we introduce the ARS bootstrap into the matrix constant false alarm rate (CFAR) method to counter the inability of the Monte Carlo methods to extract the detection threshold due to the limited number of pulses under a certain false alarm probability. We assume the echo data of each range cell in a coherent processing interval (CPI) to be a stationary time series, and regard target detection as a two-sample hypothesis testing problem. We propose a statistic to test the equivalence of the autocovariance corresponding to two potential time series by using the correlation between pulses in a short pulse dataset, and analyze the validity of the ARS bootstrap for this statistic. The statistic distribution is obtained according to the bootstrap procedure and the threshold is obtained under fixed false alarm probabilities. Numerical experiments on the simulated and real sea clutter data were conducted to compare the detection results of the partial matrix CFAR detectors for the short and long pulse cases. With the introduction of the bootstrap method, we do not need to model the clutter amplitude distribution and the related statistical characteristics during target detection. To improve the detection probability with low signal-to-clutter ratio (SCR), we propose the two-sample autocovariance vector (ACV) detector based on the proposed statistic and use the correlation properties between pulses. We observed a 28% increase in detection probability when the SCR was 0 dB compared to other matrix CFAR methods, such as detectors based on Logarithmic Euclidean (LE) distance, Kernel Function-based Symmetric Kullback-Leibler (KSKL) divergence, and Total Bregman divergence (TBD) for the short pulse case.

A Novel Unsupervised Clustering and Domain Adaptation Framework for Rotating Machinery Fault Diagnosis

Deep learning-based fault diagnosis methods require a large number of labeled datasets. However, considering the changing operating conditions, it is impractical to obtain labeled datasets for all cases. Therefore, this study proposes a new unsupervised clustering and domain adaptation framework to circumvent data deficiency and domain issues. The proposed framework comprises two steps: unsupervised clustering and domain adaptation. In the unsupervised clustering, an expectation-maximization adversarial autoencoder, which combines an expectation-maximization algorithm with an adversarial autoencoder, is used for feature extraction and subspace mapping. Subsequently, the mapped features are clustered using a Gaussian mixture model. In the domain adaptation, a domain synchronization that is based on the symmetric Kullback-Leibler divergence metric is used to infer the relationship between the source and target domain clusters. The experiments on two rolling-element-bearing datasets validate the effectiveness of our method. Specifically, our method performs domain adaptation without retraining, which is promising for real industrial applications.

A novel scheme based on information theory and transfer learning for multi classes motor imagery decoding

AbstractThe most important challenges of classifying Motor Imagery tasks based on the EEG signal are low signal‐to‐noise ratio, non‐stationarity, and the high subject dependence of the EEG signal. In this study, a framework for multi‐class decoding of Motor Imagery signals is presented. This framework is based on information theory and hybrid deep learning along with transfer learning. In this study, the OVR‐FBDiv method, which is based on the symmetric Kullback—Leibler divergence, is used to differentiate between features of different classes and highlight them. Then, the mRMR algorithm is used to select the most distinctive features obtained from the filters of symmetric KL divergence. Finally, a hybrid deep neural network consisting of CNN and LSTM is used to learn the spatial and temporal features of the EEG signal along with the transfer learning technique to overcome the problem of subject dependence in EEG signals. The average value of Kappa for the classification of 4‐class Motor Imagery data on BCI competition IV dataset 2a by the proposed method is 0.84. Also, the proposed method is compared with other state‐of‐the‐art methods.

Deconfounded recommendation via causal intervention

Traditional recommenders suffer from hidden confounding factors, leading to the spurious correlations between user/item profiles and user preference prediction, i.e., the confounding bias issue. Most works resort to only one confounding bias, which greatly block their applications on recommendations with mixture confounder, i.e., more than one bias. It is therefore of practical demand to empower the recommender with the capability of debiasing different biases from data. Moreover, the positive effect of bias is neglected in most previous works. We argue that confounding bias is actually beneficial for capturing users’ preferences in some recommendation scenarios. In this paper, we propose a novel deconfounded causal learning method called GCRec (Graph Causal Recommendetion) to debias two confounders: social network confounder and item group confounder. We employ Graph Neural Networks (GNNs) to aggregate user-user connections for social networks and user-item interactions for item groups in order to learn high-order representations that can efficiently debias these two confounders from a causal view. In the inference stage, we use symmetric Kullback–Leibler divergence to measure the user preference drift. If the divergence is large, we perform the causal intervention to alleviate the bias amplification caused by confounders on user preferences. Otherwise, we incorporate the user preferences that can potentially deliver a positive effect on favoring recommendation performance. Extensive experiments are conducted on two benchmark datasets to verify that GCRec outperforms state-of-the-art methods and achieves robust recommendations.

Unsupervised Learning Discriminative MIG Detectors in Nonhomogeneous Clutter

Principal component analysis (PCA) is a commonly used pattern analysis method that maps high-dimensional data into a lower-dimensional space maximizing the data variance, that results in the promotion of separability of data. Inspired by the principle of PCA, a novel type of learning discriminative matrix information geometry (MIG) detectors in the unsupervised scenario are developed, and applied to signal detection in nonhomogeneous environments. Hermitian positive-definite (HPD) matrices can be used to model the sample data, while the clutter covariance matrix is estimated by the geometric mean of a set of secondary HPD matrices. We define a projection that maps the HPD matrices in a high-dimensional manifold to a low-dimensional and more discriminative one to increase the degree of separation of HPD matrices by maximizing the data variance. Learning a mapping can be formulated as a two-step mini-max optimization problem in Riemannian manifolds, which can be solved by the Riemannian gradient descent algorithm. Three discriminative MIG detectors are illustrated with respect to different geometric measures, i.e., the Log-Euclidean metric, the Jensen–Bregman LogDet divergence and the symmetrized Kullback–Leibler divergence. Simulation results show that performance improvements of the novel MIG detectors can be achieved compared with the conventional detectors and their state-of-the-art counterparts within nonhomogeneous environments.

Elaboration Models with Symmetric Information Divergence.

Various statistical methodologies embed a probability distribution in a more flexible family of distributions. The latter is called elaboration model, which is constructed by choice or a formal procedure and evaluated by asymmetric measures such as the likelihood ratio and Kullback-Leibler information. The use of asymmetric measures can be problematic for this purpose. This paper introduces two formal procedures, referred to as link functions, that embed any baseline distribution with a continuous density on the real line into model elaborations. Conditions are given for the link functions to render symmetric Kullback-Leibler divergence, Rényi divergence, and phi-divergence family. The first link function elaborates quantiles of the baseline probability distribution. This approach produces continuous counterparts of the binary probability models. Examples include the Cauchy, probit, logit, Laplace, and Student-t links. The second link function elaborates the baseline survival function. Examples include the proportional odds and change point links. The logistic distribution is characterized as the one that satisfies the conditions for both links. An application demonstrates advantages of symmetric divergence measures for assessing the efficacy of covariates.

Symmetric Information–Theoretic Metric Learning for Target Detection in Hyperspectral Imagery

Metric learning-based methods, which yield great performance and show considerable potential to improve the performance of hyperspectral image processing, aim to calculate the Mahalanobis distance metric matrix. In this article, we proposed a symmetric information-theoretic metric learning (SITML) method for hyperspectral target detection. The SITML algorithm is designed based on the classical information-theoretic metric learning (ITML) and, minimizes the differential Kullback–Leibler (KL) divergence. To enhance both of the detection performance and the generalization ability, we build metric spaces from the neighborhood of training samples to preserve the local discriminative information. Then, we conduct local pairwise constraints to maximize the Jeffery divergence (also named the symmetric KL divergence) of two multivariate Gaussian distributions to solve the problem of an asymmetric KL divergence. Finally, we use a closed-form solution to solve the optimization problem. Intensive experiments on three hyperspectral datasets indicate that SITML outperforms the classical ITML algorithm and other state-of-the-art target detection methods.

Similarity Search on Wafer Bin Map Through Nonparametric and Hierarchical Clustering

Searching for and comparing similar wafer maps can provide crucial information for root cause analysis in the manufacturing process of integrated circuits. Owing to the high dimensionality and complexity of defect patterns, comparison of similar maps in their entirety is inefficient. This paper proposes an automated similarity ranking system with a novel feature set as a reduced representation of wafer maps. To detect systematic failure patterns across wafer maps, we use nonparametric Bayesian clustering based on the Dirichlet process Gaussian mixture model, and hierarchical clustering based on the symmetric Kullback–Leibler divergence. The proposed features are efficient because they require minimal computation and storage; furthermore, they allow for highly discriminative rankings of similar failure patterns. Thus they are suitable for large-scale analysis of wafer maps. The proposed method is experimentally verified using a real wafer map dataset from a semiconductor manufacturing company, and a subset of WM-811K.

Structural anomaly detection based on probabilistic distance measures of transmissibility function and statistical threshold selection scheme

As a mathematical representation of the output-to-output relationship, transmissibility function (TF) has been extensively applied in structural damage detection due to its robustness to influences of the input variations. As in most engineering fields, dealing with the problem of uncertainty in TF-based feature detection is an issue of fundamental importance. In this study, a new statistical, data-driven damage detection algorithm is proposed by rigorously modelling the variability of TF without postprocessing with circularly-symmetric complex Gaussian ratio distribution. The probabilistic distance of Symmetric Kullback-Leibler (SKL) divergence between TFs under baseline condition and potential damage scenarios which can measure the dissimilarity of probability distributions for the TFs under different states are computed as a damage index (DI) to detect structural anomaly. Compared against Mahalanobis distance which has the implicit assumption that the normal condition set is governed by Gaussian statistics, the probabilistic distance measure proposed in this study can deal with the deviations in TFs not following Gaussian distribution. A statistically rigorous threshold selection scheme integrating Bayesian inference strategy and Monte Carlo discordancy test is proposed to detect the the presence of damage by accommodating the uncertainties of measurements and the probabilistic model of TF. Numerical, experimental, and field test studies are conducted to validate the potential of probabilistic distance measure of TFs in anomaly detection under ambient vibration instead of forced vibration testing. Results demonstrate satisfactory performance of the proposed approach for detecting the existence and quantify the relative damage severity from a global perspective.

A sequential multi-prior integration and updating method for complex multi-level system based on Bayesian melding method

In engineering, there exist multiple priors about system and subsystems uncertainties, which should be integrated properly to analyze the system reliability. In the past research, an iterative updating procedure based on Bayesian Melding Method (I-BMM) was developed to merge and update multiple priors for the double-level system. However, the in-depth study in this paper shows that the original iterative procedure has no effect on the prior updating. Thus it is proposed that only a single BMM iteration process is needed following the original prior integration and updating formulation. BMM involves the sampling procedure for the probability density function (PDF) updating, wherein it is generally difficult to define the sampling number properly for obtaining accurate priors. To address this problem, a sequential prior integration and updating framework based on the original single BMM iteration process (S-BMM) is developed in this paper. In each cycle of prior updating, the sample number is sequentially added, and the difference between prior distributions obtained in the two consecutive cycles is measured with the symmetric Kullback-Leibler Divergence (SKLD). The sequential procedure is continued until the convergence to the accurate updated prior. The S-BMM framework for double-level systems is further extended for multi-level systems. Situations with some missing subsystem or component priors are also discussed. Finally, two numerical examples and one satellite engineering case are used to demonstrate and verify the proposed algorithms.

Functional Clustering on a Circle Using von Mises Mixtures

This paper addresses the question of clustering density curves around a unit circle by approximating each such curve by a mixture of an appropriate number of von Mises distributions. This is done first by defining a distance between any two such curves either via \(L^2\) or a symmetrized Kullback–Leibler divergence. We show that both these measures yield similar results. After demonstrating via simulations that the proposed clustering methods work successfully, they are applied on an illustrative sample of Optical Coherence Tomography data.

A just-in-time modeling approach for multimode soft sensor based on Gaussian mixture variational autoencoder

Industrial data are often high-dimensional, nonlinear and multiple-modal. This paper develops a soft sensor model based on Gaussian mixture Variational Autoencoder (GMVAE) under the just-in-time learning (JITL) framework. To extract latent representations with multimode characteristics, GMVAE as a deep neural network model is utilized by considering Gaussian mixture models (GMM) in the latent space. After training the GMVAE model, each latent (or feature) variable can be described through a Gaussian mixture distribution. Subsequently, when a new sample arrives, a mixture symmetric Kullback-Leibler (MSKL) divergence is utilized to measure its similarity with historical data samples. MSKL divergence can measure similarity between two Gaussian mixture probability density functions. Based on the MSKL divergence, weighted input and output historical data are obtained, and then a local model is established. The effectiveness of the proposed soft sensor modeling method is validated through a numerical example along with simulation on the Tennessee Eastman benchmark process.

An iterative information integration method for multi-level system reliability analysis based on Bayesian Melding Method

For uncertainty modeling and reliability analysis of a complex system, there generally exists multi-source information, which should be fully considered and used synthetically. Research shows that the Bayesian Melding Method (BMM) is a useful tool to merge the multi-source information. However, how to apply BMM for the complex system with a multi-level hierarchical structure remains a challenging issue. To address this problem, this paper proposes an iterative information integration method for multi-level system structures so as to fully integrate the information between different levels. A complete single iteration consists of the updating process from the system bottom to top level and then from the system top to bottom level. To facilitate the updating process, the complex multi-level system is first decomposed into several basic double-level units, within which the information integration can be conveniently conducted with the proposed discrete or continuous BMM methods. To check the iteration convergence, the symmetric Kullback-Leibler Divergence (SKLD) is adopted to measure the difference between the updated system distributions obtained in the two consecutive iteration processes.Finally, three case studies with discrete and continuous information integration problems are used to demonstrate and validate the proposed method.

Visualizing probabilistic models in Minkowski space with intensive symmetrized Kullback-Leibler embedding

We show that the predicted probability distributions for any $N$-parameter statistical model taking the form of an exponential family can be explicitly and analytically embedded isometrically in a $N{+}N$-dimensional Minkowski space. That is, the model predictions can be visualized as control parameters are varied, preserving the natural distance between probability distributions. All pairwise distances between model instances are given by the symmetrized Kullback-Leibler divergence. We give formulas for these intensive symmetrized Kullback Leibler (isKL) coordinate embeddings, and illustrate the resulting visualizations with the Bernoulli (coin toss) problem, the ideal gas, $n$ sided die, the nonlinear least squares fit, and the Gaussian fit. We highlight how isKL can be used to determine the minimum number of parameters needed to describe probabilistic data, and conclude by visualizing the prediction space of the two-dimensional Ising model, where we examine the manifold behavior near its critical point.

A mutual information-based Variational Autoencoder for robust JIT soft sensing with abnormal observations

Considering industrial process with high-dimensional, intrinsic nonlinearities and possibly abnormal observations, a robust deep learning soft sensor model is developed under the just-in-time learning framework. As an unsupervised deep learning approach, Variational Autoencoder (VAE) has been successfully applied to soft sensing problems owing to its ability to describe the latent representations by probability distributions. In this work, to construct high performance soft sensor model, mutual information (MI) is first introduced for input variable selection. By further incorporating MI as weights on variable of the traditional VAE model, a MI-based output-relevant VAE is developed. For each new sample that arrives, by utilizing Symmetric Kullback-Leibler (SKL) divergence, its relevance with historical samples is determined. Based on the SKL divergence, the input samples that are most relevant to the query sample can be collected. The selected historical input samples and corresponding output samples are employed to build a Gaussian process regression (GPR) local model. Expectation maximation (EM) algorithm is utilized to deal with the nonlinearity and abnormal output observations in GPR local model simultaneously for robustness of the soft sensors. Numerical simulations and a benchmark process are employed to validate the effectiveness of the proposed soft sensor, which demonstrates its superior performance over traditional approaches.

Hierarchical Clustering of Materials With Defects Using Impact-Echo Testing

Signals obtained from impact-echo techniques can be used to detect and classify the defects in damaged materials. The defects change the wave propagation between the impact and the sensors producing particular spectrum elements, which define the feature vector. We propose a hierarchical clustering method that models the feature vector as a mixture of Gaussians (MoG) for every class and then merges different clusters using as a distance measure the symmetric Kullback-Leibler (KL) divergence. Since there is no closed-form solution to the KL divergence between MoGs, some approximations are introduced. We apply the hierarchical clustering algorithms to the signals obtained from real specimens made of aluminum alloy. The samples are classified into four classes according to the state: homogeneous (no defect), one hole, one crack, and multiple defects. We compare the performance of different approximations and discuss the dendrograms that are obtained. Similar kinds of defects are clustered first, and more importantly, the high-level hierarchy is able to distinguish between the defective and nondefective materials.

Speech enhancement using adaptive thresholding based on gamma distribution of Teager energy operated intrinsic mode functions

This paper introduces a new speech enhancement algorithm based on the adaptive threshold of intrinsic mode functions (IMFs) of noisy signal frames extracted by empirical mode decomposition. Adaptive threshold values are estimated by using the gamma statistical model of Teager energy operated IMFs of noisy speech and estimated noise based on symmetric Kullback–Leibler divergence. The enhanced speech signal is obtained by a semisoft thresholding function, which is utilized by threshold IMF coefficients of noisy speech. The method is tested on the NOIZEUS speech database and the proposed method is compared with wavelet-shrinkage and EMD-shrinkage methods in terms of segmental SNR improvement (SegSNR), weighted spectral slope (WSS), and perceptual evaluation of speech quality (PESQ). Experimental results show that the proposed method provides a higher SegSNR improvement in dB, lower WSS distance, and higher PESQ scores than wavelet-shrinkage and EMD-shrinkage methods. The proposed method shows better performance than traditional threshold-based speech enhancement approaches from high to low SNR levels.

Design and Statistical Analysis of Tapered Coprime and Nested Arrays for the Min Processor

Coprime and nested arrays are sparse sensor arrays that provide $\mathcal {O}(MN)$ degrees of freedom using only $\mathcal {O}(M+N)$ sensors. The signals sampled by these sparse arrays contain the aliasing artifact; the min processor is used to disambiguate this aliasing artifact. The min processor beampattern has the same main lobe width and resolution as the beampattern of a full uniform linear array (ULA) with many more sensors. However, the peak side lobe (PSL) height of the min processor beampattern is higher than the PSL height of a full ULA with the same taper and resolution if the sparse arrays are not extended. Extending the sparse arrays, while keeping the intersensor spacing fixed, can reduce the PSL height to the level of the full ULA. Until now, the analytical expressions of the extension factor required to match the PSL height have not been found for the min processor. Also, the analytical expressions of the probability density function (PDF), mean, and variance of the min processor output for non-uniform tapers have not been derived. In this paper, we derive both of these analytical expressions. We consider the uniform, Hamming, Hann, Blackman-Harris, and Dolph-Chebyshev tapers for both coprime and nested arrays. We use the derived PDF in evaluating detection performance metrics such as receiver operation characteristic, Kullback Leibler divergence, and symmetric Kullback Leibler divergence. Our results show that min processing is superior to conventional beamforming and product processing in Gaussian signal detection for a given extended coprime or nested array.