Kullback-Leibler Research Articles

SNOOKER: a dataset generator for helpdesk services

AbstractThe availability of suitable datasets and data generators is crucial for developing intelligent systems, especially in helpdesk services. However, the lack of publicly accessible data generators focused on helpdesk operations, where incidents are often closed without detailing the treatment procedures, poses challenges to implementing intelligent systems such as recommender systems. To address this issue, a dataset generator can be employed to simulate helpdesk incidents. This paper introduces SNOOKER (dataSet geNeratOr fOr helpdesK sERvices), a customizable dataset generator designed to create and treat helpdesk tickets, including domain-specific incidents (e.g., cybersecurity) by orchestrating simulated actions and multiple IT teams. SNOOKER’s output is compared against a real anonymized dataset from S21Sec Cyber Solutions by Thales. The datasets are evaluated using Kolmogorov-Smirnov, Kullback-Leibler Divergence, and Hellinger distance tests, with results indicating similar distributions. For example, the first metric returned a low K-S value and a p-value exceeding 5%, while the second and third measures presented 0.003 and 0.03, respectively. Furthermore, experiments with different team configurations revealed that ticket scheduling highly depends on each team’s operators’ numbers and work shifts, increasing with unbalanced shifts and fewer operators.

Deep Learning-Based In Situ Micrograph Synthesis and Augmentation for Crystallization Process Image Analysis

Deep learning-based in situ imaging and analysis for crystallization process are essential for optimizing product qualities, reducing experimental costs through real-time monitoring, and controlling the process. However, large and high-quality annotated datasets are required to train accurate models, which are time consuming. Therefore, we proposed a novel methodology that applied image synthesis neural networks to generate virtual information-rich images, enabling efficient and rapid dataset expansion while simultaneously reducing annotation costs. Experiments were conducted on the L-alanine crystallization process to obtain process images and to validate the proposed workflow. The proposed method, aided by interpolation augmentation and data warping augmentation to enhance data richness, utilized only 25% of the training annotations, consistently segmenting crystallization process images comparable to those models utilizing 100% of the training data annotations, achieving an average precision of nearly 98%. Additionally, based on the analysis of Kullback–Leibler divergence, the proposed method demonstrated excellent performance in extracting in situ information regarding aspect ratios and crystal size distributions during the crystallization process. Moreover, its ability to leverage expert labels with a four-fold enhanced efficiency holds great potential for advancing various applications in crystallization processes.

Characteristics of the GLLE Family of Lifetime Distributions, and Applications

ABSTRACT. A Generalized Log-Logistic (GLL) family of lifetime distributions is one in which any pair of distributions are related through a GLL transformation, for some (non-negative) value of the transformation parameter (the odds function of the second distribution is the power of the odds function of the first distribution). We consider GLL families generated from an exponential distribution. We derive some useful characteristics of these distributions; moreover, we discuss the hazard rates and Kullback-Leibler divergence, then illustrate the usefulness of this distribution family by fitting real data sets.

Distributions and Correlation Properties of Offshore Wind Speeds and Wind Speed Increments

We determine distributions and correlation properties of offshore wind speeds and wind speed increments by analyzing wind data sampled with a resolution of one second for 20 months at different heights above sea level in the North Sea. Distributions of horizontal wind speeds can be fitted to Weibull distributions with shape and scale parameters varying weakly with the vertical height separation. Kullback–Leibler divergences between distributions at different heights change with the squared logarithm of the height ratio. Cross-correlations between time derivatives of wind speeds are long-term anticorrelated, and the even parts of their correlation functions satisfy sum rules. Distributions of horizontal wind speed increments change from a tent-like shape to a Gaussian with rising increment lag. A surprising peak occurs in the left tail of the increment distributions for lags in a range 10-200km\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$10-200\\,{km}$$\\end{document} after applying the Taylor’s hypothesis locally to transform time lags into distances. The peak is decisive in order to obtain an expected and observed linear scaling of third-order structure functions with distance. This suggests that it is an intrinsic feature of atmospheric turbulence.

Reducing electrocardiographic interference in the multichannel electromyogram to help muscle fatigue assessment in low-intensity contractions.

Surface electromyography (EMG) can be used in the rehabilitation of musculoskeletal disorders, for evaluating the coordination of muscles that stabilize a joint, or as an objective tool in assessing muscle fatigue. This latter may be achieved by evaluating the low-frequency content of the EMG. However, the electrocardiogram (ECG) interference is also recorded when EMG is acquired close to the heart. It may mask or even modify the information of interest in the EMG. Different signal processing techniques have been proposed to eliminate ECG artifacts from the EMG signals, the high-pass filter being the most used one. Nevertheless, this classic filtering approach would also attenuate EMG activities below 30 Hz, which contain information from low-intensity muscle contractions. This work addresses the automatic ECG attenuation when multichannel EMG is collected on the left pectoralis major muscle during a muscle fatigue test. The proposed ECG artifact mitigation extends a successful automatic template subtraction (TS) approach, which does not require an extra reference channel and is now applied to independent EMG source signal components extracted using a blind source separation technique (ICA, Independent Component Analysis). The automatic detection of ECG components was performed through two alternative measures: entropy and Kullback-Leibler divergence. While the ECG interference seems to hamper the detection of muscle fatigue in the low-contraction regime, the association ICA+TS preserved better the EMG low-frequency content, and the entropy-based automatic detection was found to be more suitable, avoiding possible errors that might arise from manual detection procedures.

Enhanced Kullback–Leibler Divergence Based Pilot Protection for Lines Connecting Battery Energy Storage Stations

In light of the challenges faced by existing protection methods when battery energy storage station (BESS) and various nonideal conditions emerge, enhanced Kullback–Leibler divergence-based time-domain pilot protection is put forward in this article. Given the discrepancy in current waveforms between internal and external faults, KLD is employed to quantify the degree of similarity between two current waveforms, enabling the accurate distinction between internal and external faults. To eliminate the asymmetry of KLD, improved KLD(IKLD) is introduced. Whereafter, a technique is proposed to convert the current sampling sequence into a probability distribution that can be utilized by IKLD. To counteract the negative influence of current transformer(CT) saturation on IKLD-based protection, the time-domain waveform features of summation current are adequately employed to develop an enhanced KLD(EKLD). Further, high and low thresholds are legitimately designed to ensure both dependability and security. The flowchart of EKLD-based pilot protection is described, and its effectiveness is extensively validated by variation in fault resistance, location, type and operating mode of BESS using PSCAD and a hardware-in-the-loop(HIL) testing platform. Additionally, proposed protection outclasses power frequency component-based current differential protection and some up-to-date presented time-domain protection methods.

Deep Multilayer Sparse Regularization Time- Varying Transfer Learning Networks With Dynamic Kullback–Leibler Divergence Weights for Mechanical Fault Diagnosis

Deep Multilayer Sparse Regularization Time- Varying Transfer Learning Networks With Dynamic Kullback–Leibler Divergence Weights for Mechanical Fault Diagnosis

Just-in-time framework for robust soft sensing based on robust variational autoencoder

Modeling with high-dimensional data subject to abnormal observations have always been a practical interest. In this paper, under the just-in-time learning (JITL) framework, a robust soft sensor modeling approach is developed based on robust Variational Autoencoder (VAE). Unlike the vanilla VAE that extracts features from the given dataset under the Gaussian prior assumption, robust VAE employs Student’s t-distribution as prior distribution to handle abnormal data. Under assumption of the Student’s t-prior, the proposed robust VAE model is capable of describing collected data contaminated with outliers. Once the robust VAE model is trained, each robust feature variable in the latent space can be determined. Subsequently, similarity measure is calculated using robust Kullback-Leibler divergence between two Student’s t-distributions, that is, the distribution of a new data sample and that of each historical data sample. After completing similarity measurement for a query sample, the weights for input-output historical data can be determined. Based on these weighted historical data samples, a robust probabilistic principal component regression (PPCR) is utilized to perform local modeling for prediction. Numerical simulations, including the Tennessee Eastman and Penicillin fermentation benchmark processes, are utilized to validate the proposed JITL-based robust soft sensor modeling method.

Coherent Set Identification Via Direct Low Rank Maximum Likelihood Estimation

We analyze connections between two low rank modeling approaches from the last decade for treating dynamical data. The first one is the coherence problem (or coherent set approach), where groups of states are sought that evolve under the action of a stochastic transition matrix in a way maximally distinguishable from other groups. The second one is a low rank factorization approach for stochastic matrices, called direct Bayesian model reduction (DBMR), which estimates the low rank factors directly from observed data. We show that DBMR results in a low rank model that is a projection of the full model, and exploit this insight to infer bounds on a quantitative measure of coherence within the reduced model. Both approaches can be formulated as optimization problems, and we also prove a bound between their respective objectives. On a broader scope, this work relates the two classical loss functions of nonnegative matrix factorization, namely the Frobenius norm and the generalized Kullback–Leibler divergence, and suggests new links between likelihood-based and projection-based estimation of probabilistic models.

Pattern detection in the vehicular activity of bus rapid transit systems

In this paper, we explore different methods to detect patterns in the activity of bus rapid transit (BRT) systems focusing on two aspects of transit: infrastructure and the movement of vehicles. To this end, we analyze records of velocity and position of each active vehicle in nine BRT systems located in the Americas. We detect collective patterns that characterize each BRT system obtained from the statistical analysis of velocities in the entire system (global scale) and at specific zones (local scale). We analyze the velocity records at the local scale applying the Kullback-Leibler divergence to compare the vehicular activity between zones. This information is organized in a similarity matrix that can be represented as a network of zones. The resulting structure for each system is analyzed using network science methods. In particular, by implementing community detection algorithms on networks, we obtain different groups of zones characterized by similarities in the movement of vehicles. Our findings show that the representation of the dataset with information of vehicles as a network is a useful tool to characterize at different scales the activity of BRT systems when geolocalized records of vehicular movement are available. This general approach can be implemented in the analysis of other public transportation systems.

Visualizing departures from marginal homogeneity for square contingency tables with ordered categories

Square contingency tables are a special case commonly used in various fields to analyze categorical data. Although several analysis methods have been developed to examine marginal homogeneity (MH) in these tables, existing measures are single-summary ones. To date, a visualization approach has yet to be proposed to intuitively depict the results of MH analysis. Current measures used to assess the degree of departure from MH are based on entropy such as the Kullback-Leibler divergence and do not satisfy distance postulates. Hence, the current measures are not conducive to visualization. Herein we present a measure utilizing the Matusita distance and introduce a visualization technique that employs sub-measures of categorical data. Through multiple examples, we demonstrate the meaningfulness of our analytical framework approach and validate its usefulness to provide insightful interpretations.

A nonlinear filter based on GSK for relative navigation using relative orbital elements

Due to the sensitivity of the relative orbital elements model to the measurement noise, the non-stationary heavy-tailed noise(NSHT) induced by the time-varying environment during the relative navigation usually leads to filter divergence. To address this problem, a new nonlinear filter based on Gaussian-Student's-Multivariate K(GSK) mixture distribution is proposed in this paper. A Dirichlet stochastic mixture vector fusing Gaussian, Student's t, and Multivariate K distributions is introduced, thus proposing a GSK mixture distribution modeling measurement likelihood; then the Kullback-Leibler Divergence (KLD) of the true posteriori probability density function(PDF) and the approximate posteriori PDF are minimized by a variational Bayesian(VB) technique to solve for the state and parameter approximate a posteriori estimations, and finally a new nonlinear filter based on the GSK mixture distribution is derived for angles-only relative navigation in time-varying environments. Simulation outcomes indicate that the filter can realize state estimation in non-stationary states effectively with 45.16% higher estimation accuracy than the existing advanced filters.

Some Improvements on Good Lattice Point Sets

Good lattice point (GLP) sets are a type of number-theoretic method widely utilized across various fields. Their space-filling property can be further improved, especially with large numbers of runs and factors. In this paper, Kullback-Leibler (KL) divergence is used to measure GLP sets. The generalized good lattice point (GGLP) sets obtained from linear-level permutations of GLP sets have demonstrated that the permutation does not reduce the criterion maximin distance. This paper confirms that linear-level permutation may lead to greater mixture discrepancy. Nevertheless, GGLP sets can still enhance the space-filling property of GLP sets under various criteria. For small-sized cases, the KL divergence from the uniform distribution of GGLP sets is lower than that of the initial GLP sets, and there is nearly no difference for large-sized points, indicating the similarity of their distributions. This paper incorporates a threshold-accepting algorithm in the construction of GGLP sets and adopts Frobenius distance as the space-filling criterion for large-sized cases. The initial GLP sets have been included in many monographs and are widely utilized. The corresponding GGLP sets are partially included in this paper and will be further calculated and posted online in the future. The performance of GGLP sets is evaluated in two applications: computer experiments and representative points, compared to the initial GLP sets. It shows that GGLP sets perform better in many cases.

Enhanced predictive PDF control of stochastic distribution systems with neural network compensation and its application

Compared to the conventional control algorithms that use mean and variance as indicators, predictive probability density function (PDF) control can effectively handle the output PDF control problem of non-Gaussian stochastic distribution systems. However, the existing predictive PDF control method does not consider the construction error between output PDF and weight, thus the control performance is still unsatisfactory. Therefore, this paper proposes a new Enhanced Predictive PDF control method (En-PDF) to improve the output PDF control performance of stochastic distribution systems. The proposed method mainly consists of two parts: the predictive PDF control part and the neural network compensation control part aiming to reduce the bias of output PDF. First, the Radical Basis Functions (RBFs) are used to approximate the PDF of the stochastic systems output, and then a prediction model representing the relationship between input and weight is established using the subspace identification algorithm to design the predictive PDF control for the stochastic systems. Next, the Kullback-Leibler (KL) divergence is used to measure the similarity between the output PDF and the set PDF, combined with the weight error and compensation to design a new performance index. Based on this, the parameters of the neural network are adjusted using the gradient descent algorithm to obtain the optimal compensation, and the stability and tracking performance of the proposed algorithm are analyzed using inductive reasoning method. Finally, the predictive PDF control input with the compensation work together on the controlled plant to achieve high-performance control of the output PDF of non-Gaussian stochastic distribution systems. Both simulation experiments and physical control experiments validate the effectiveness and superiority of the proposed method.

Evaluating model fit for type II censored data: a Bayesian non-parametric approach based on the Kullback-Leibler divergence estimation

Model checking evaluates the appropriateness of a statistical model based on the observed data, and it is essential to make valid statistical analyses. In this paper, a new procedure for model checking type II censored data is proposed. This procedure combines the Kullback-Leibler divergence, the Dirichlet process, and the relative belief ratio. The method is implemented via a computational algorithm and is explained through several examples.

Asymptotic Properties of a Statistical Estimator of the Jeffreys Divergence: The Case of Discrete Distributions

We investigate the asymptotic properties of the plug-in estimator for the Jeffreys divergence, the symmetric variant of the Kullback–Leibler (KL) divergence. This study focuses specifically on the divergence between discrete distributions. Traditionally, estimators rely on two independent samples corresponding to two distinct conditions. However, we propose a one-sample estimator where the condition results from a random event. We establish the estimator’s asymptotic unbiasedness (law of large numbers) and asymptotic normality (central limit theorem). Although the results are expected, the proofs require additional technical work due to the randomness of the conditions.

Discounted fully probabilistic design of decision rules

Axiomatic fully probabilistic design (FPD) of optimal decision rules strictly extends the decision making (DM) theory represented by Markov decision processes (MDP). This means that any MDP task can be approximated by an explicitly found FPD task whereas many FPD tasks have no MDP equivalent. MDP and FPD model the closed loop — the coupling of an agent and its environment — via a joint probability density (pd) relating the involved random variables, referred to as behaviour. Unlike MDP, FPD quantifies agent's aims and constraints by an ideal pd. The ideal pd is high on the desired behaviours, small on undesired behaviours and zero on forbidden ones. FPD selects the optimal decision rules as the minimiser of Kullback-Leibler's divergence of the closed-loop-modelling pd to its ideal twin. The proximity measure choice follows from the FPD axiomatics.MDP minimises the expected total loss, which is usually the sum of discounted partial losses. The discounting reflects the decreasing importance of future losses. It also diminishes the influence of errors caused by:▪ the imperfection of the employed environment model;▪ roughly-expressed aims;▪ the approximate learning and decision-rules design.The established FPD cannot currently account for these important features. The paper elaborates the missing discounted version of FPD. This non-trivial filling of the gap in FPD also employs an extension of dynamic programming, which is of an independent interest.

Feature Vector Effectiveness Evaluation for Pattern Selection in Computational Lithography

Pattern selection is crucial for optimizing the calibration process of optical proximity correction (OPC) models in computational lithography. However, it remains a challenge to achieve a balance between representative coverage and computational efficiency. This work presents a comprehensive evaluation of the feature vectors’ (FVs’) effectiveness in pattern selection for OPC model calibration, leveraging key performance indicators (KPIs) based on Kullback–Leibler divergence and distance ranking. Through the construction of autoencoder-based FVs and fast Fourier transform (FFT)-based FVs, we compare their efficacy in capturing critical pattern features. Validation experimental results indicate that autoencoder-based FVs, particularly augmented with the lithography domain knowledge, outperform FFT-based counterparts in identifying anomalies and enhancing lithography model performance. These results also underscore the importance of adaptive pattern representation methods in calibrating the OPC model with evolving complexities.

Remote Sensing Image Segmentation Based on Probabilistic Fuzzy Local Information Clustering

Abstract. In this paper, a remote sensing image segmentation based on probabilistic fuzzy local information clustering algorithm is proposed. First, assuming that the spectral measure within the same ground object follows the same probability distribution. The dissimilarity between pixel and object is modeled by the negative logarithm of the Gaussian probability density function. It can improve the noise sensitive problem caused by the Euclidean distance which can only describe the data with isotropic distribution. Then, in order to consider the effect of local spatial constraint, on the one hand, the probability measure is used to modify the local fuzzy factor to establish the dissimilarity measure with spatial constraints. On the other hand, the hidden Markov random field is used to model the prior probability model of pixel membership. Next, the entropy regularization term of the objective function is built by combining the Kullback-Leibler(KL) maximum entropy model to further improve the robustness and noise resistance. The qualitative and quantitative analysis of simulated image and different types of real remote sensing images show that the proposed algorithm can effectively overcome the above problems and further improve the accuracy of image segmentation to over 95%.

Local inconsistency detection using the Kullback–Leibler divergence measure

BackgroundThe standard approach to local inconsistency assessment typically relies on testing the conflict between the direct and indirect evidence in selected treatment comparisons. However, statistical tests for inconsistency have low power and are subject to misinterpreting a p-value above the significance threshold as evidence of consistency.MethodsWe propose a simple framework to interpret local inconsistency based on the average Kullback–Leibler divergence (KLD) from approximating the direct with the corresponding indirect estimate and vice versa. Our framework uses directly the mean and standard error (or posterior mean and standard deviation) of the direct and indirect estimates obtained from a local inconsistency method to calculate the average KLD measure for selected comparisons. The average KLD values are compared with a semi-objective threshold to judge the inconsistency as acceptably low or material. We exemplify our novel interpretation approach using three networks with multiple treatments and multi-arm studies.ResultsAlmost all selected comparisons in the networks were not associated with statistically significant inconsistency at a significance level of 5%. The proposed interpretation framework indicated 14%, 66%, and 75% of the selected comparisons with an acceptably low inconsistency in the corresponding networks. Overall, information loss was more notable when approximating the posterior density of the indirect estimates with that of the direct estimates, attributed to indirect estimates being more imprecise.ConclusionsUsing the concept of information loss between two distributions alongside a semi-objectively defined threshold helped distinguish target comparisons with acceptably low inconsistency from those with material inconsistency when statistical tests for inconsistency were inconclusive.