Kullback-Leibler Divergence Measure Research Articles

Detecting drifts in data streams using Kullback-Leibler (KL) divergence measure for data engineering applications

AbstractThe exponential growth of data coupled with the widespread application of artificial intelligence(AI) presents organizations with challenges in upholding data accuracy, especially within data engineering functions. While the Extraction, Transformation, and Loading process addresses error-free data ingestion, validating the content within data streams remains a challenge. Prompt detection and remediation of data issues are crucial, especially in automated analytical environments driven by AI. To address these issues, this study focuses on detecting drifts in data distributions and divergence within data fields processed from different sample populations. Using a hypothetical banking scenario, we illustrate the impact of data drift on automated decision-making processes. We propose a scalable method leveraging the Kullback-Leibler (KL) divergence measure, specifically the Population Stability Index (PSI), to detect and quantify data drift. Through comprehensive simulations, we demonstrate the effectiveness of PSI in identifying and mitigating data drift issues. This study contributes to enhancing data engineering functions in organizations by offering a scalable solution for early drift detection in data ingestion pipelines. We discuss related research works, identify gaps, and present the methodology and experiment results, underscoring the importance of robust data governance practices in mitigating risks associated with data drift and improving data observability.

Quantile-based cumulative Kullback-Leibler divergence in past lifetime: Some properties and applications

This article introduces a quantile-based cumulative Kullback-Leibler (KL) divergence measure for past lifetimes (QPCKL divergence) and studies its various properties. Unlike the cumulative KL divergence using the distribution function, the proposed measure offers an alternative method for computing the divergence between two random variables, especially when one lacks a closed-form distribution function but possesses a closed-form quantile function. A characterization property of a distribution is derived using the constant QPCKL divergence model. Subsequently, a comparison of some important reliability characteristics using the proposed measure is discussed. The QPCKL divergence for a general family of transformation models is derived. We also propose a non parametric estimator for the measure, and simulation studies are carried out for validation. Finally, the importance of the estimator is also illustrated through real data sets useful in survival studies.

Image Quality Assessment: Measuring Perceptual Degradation via Distribution Measures in Deep Feature Spaces.

This study aims to develop advanced and training-free full-reference image quality assessment (FR-IQA) models based on deep neural networks. Specifically, we investigate measures that allow us to perceptually compare deep network features and reveal their underlying factors. We find that distribution measures enjoy advanced perceptual awareness and test the Wasserstein distance (WSD), Jensen-Shannon divergence (JSD), and symmetric Kullback-Leibler divergence (SKLD) measures when comparing deep features acquired from various pretrained deep networks, including the Visual Geometry Group (VGG) network, SqueezeNet, MobileNet, and EfficientNet. The proposed FR-IQA models exhibit superior alignment with subjective human evaluations across diverse image quality assessment (IQA) datasets without training, demonstrating the advanced perceptual relevance of distribution measures when comparing deep network features. Additionally, we explore the applicability of deep distribution measures in image super-resolution enhancement tasks, highlighting their potential for guiding perceptual enhancements. The code is available on website. (https://github.com/Buka-Xing/Deep-network-based-distribution-measures-for-full-reference-image-quality-assessment).

Properties and Applications of the Type I Half-logistic Nadarajah-Haghighi Distribution

A new three-parameter distribution called the type I half-logistic Nadarajah-Haghighi (T IHL N H ) is proposed. We discussed some important mathematical and statistical properties of the new model such as an explicit form of its r th moment, mean deviations,quantile function, Bonferroni and Lorenz curves. The Shannon entropy and Renyi entropy are computed, the expression for the Kullback-Leibler divergence measure is provided. The model parameters estimation was approached by the maximum likelihoodestimation (MLE), and the information matrix is obtained. The finite sample properties of the MLEs are investigated numerically by simulation studies; by examining the bias and mean square error of the estimators, and the results was satisfactory. We used tworeal data applications to demonstrate the superior performance of the T IHL N H in terms of fit over some other existing lifetime models.

The role of hyper-parameters in relational topic models: Prediction capabilities vs topic quality

In this paper, we investigate the impact of optimal hyper-parameter configuration in relational topic models. The main goal is to validate the hypothesis that single-objective Bayesian Optimization (BO) can discover a hyper-parameter setting that leads a set of relational topic models to simultaneously ensure good prediction capabilities and significant topics from a qualitative perspective. Our research, as a result of a comparative analysis performed on 7 state-of-the-art models, 5 performance measures and 3 datasets, has highlighted three main findings: (1) the majority of relational topic models are not able to offer a good trade-off between classification capabilities and topic interpretability; (2) single-objective optimization of hyper-parameters, targeted on maximizing the F1-Measure, is able to create topics that are also optimal with respect to the Kullback Leibler divergence measure; (3) the Pareto frontiers across several performance metrics reveals that the most promising trade-off between the performance metrics can be obtained by Constrained Relational Topic Models.

Distributional Validation of Precipitation Data Products with Spatially Varying Mixture Models

The high mountain regions of Asia contain more glacial ice than anywhere on the planet outside of the polar regions. Because of the large population living in the Indus watershed region who are reliant on melt from these glaciers for fresh water, understanding the factors that affect glacial melt along with the impacts of climate change on the region is important for managing these natural resources. While there are multiple climate data products (e.g., reanalysis and global climate models) available to study the impact of climate change on this region, each product will have a different amount of skill in projecting a given climate variable, such as precipitation. In this research, we develop a spatially varying mixture model to compare the distribution of precipitation in the High Mountain Asia region as produced by climate models with the corresponding distribution from in situ observations from the Asian Precipitation—Highly Resolved Observational Data Integration Towards Evaluation (APHRODITE) data product. Parameter estimation is carried out via a computationally efficient Markov chain Monte Carlo algorithm. Each of the estimated climate distributions from each climate data product is then validated against APHRODITE using a spatially varying Kullback–Leibler divergence measure. Supplementary materials accompanying this paper appear online.

A speech enhancement algorithm based on a non-negative hidden Markov model and Kullback-Leibler divergence

In this paper, we propose a supervised single-channel speech enhancement method that combines Kullback-Leibler (KL) divergence-based non-negative matrix factorization (NMF) and a hidden Markov model (NMF-HMM). With the integration of the HMM, the temporal dynamics information of speech signals can be taken into account. This method includes a training stage and an enhancement stage. In the training stage, the sum of the Poisson distribution, leading to the KL divergence measure, is used as the observation model for each state of the HMM. This ensures that a computationally efficient multiplicative update can be used for the parameter update of this model. In the online enhancement stage, a novel minimum mean square error estimator is proposed for the NMF-HMM. This estimator can be implemented using parallel computing, reducing the time complexity. Moreover, compared to the traditional NMF-based speech enhancement methods, the experimental results show that our proposed algorithm improved the short-time objective intelligibility and perceptual evaluation of speech quality by 5% and 0.18, respectively.

The stereographic generalized inverse Weibull distribution

When the unit circle is taken as a sample space, the circular data manifest in different aspects of real life. Minh and Farnum (2003) used a stereographic projection to generate new probability distributions on real line [1]. Here, we present an asymmetric distribution called Stereographic generalized inverse Weibull distribution (SGIWD) to represent circular data with focus on the inverse Stereographic Projection. The graph of probability density function is done. We consider here some statistical properties of SGIW distribution, survival and reverse hazard functions, the rth raw moments function, trigonometric moments, stress-strength reliability model, Tsallis, Renyi Shannon entropies and the Kullback–Leibler divergence measure. A simulation study is conducted to evaluate the behavior of the maximum likelihood method to estimate parameters of SGIWD.

Insect Pest Image Recognition: A Few-Shot Machine Learning Approach including Maturity Stages Classification

Recognizing insect pests using images is an important and challenging research issue. A correct species classification will help choosing a more proper mitigation strategy regarding crop management, but designing an automated solution is also difficult due to the high similarity between species at similar maturity stages. This research proposes a solution to this problem using a few-shot learning approach. First, a novel insect data set based on curated images from IP102 is presented. The IP-FSL data set is composed of 97 classes of adult insect images, and 45 classes of early stages, totalling 6817 images. Second, a few-shot prototypical network is proposed based on a comparison with other state-of-art models and further divergence analysis. Experiments were conducted separating the adult classes and the early stages into different groups. The best results achieved an accuracy of 86.33% for the adults, and 87.91% for early stages, both using a Kullback–Leibler divergence measure. These results are promising regarding a crop scenario where the more significant pests are few and it is important to detect them at earlier stages. Further research directions would be in evaluating a similar approach in particular crop ecosystems, and testing cross-domains.

On differential Renyi's–Tsallis divergence measure and its applications

In this article, we introduce the differential Renyi's–Tsallis divergence as a measure of difference between two probability distributions which implies that the proposed divergence measure is a positive constant time the sum of Renyi's and Tsallis differential divergences and also generalized the differential Kullback–Leibler divergence measure. The properties of the proposed measure are presented. Then some bounds for this divergence measure are obtained, and a new divergence measure for any distribution belonging to the exponential families is defined. Also, we present the differential Renyi's–Tsallis divergence between the order statistics. Finally, this new divergence measure is applied to the proportional reversed hazards model and testing exponentiality. We construct a goodness-of-fit test for the exponential distribution based on the estimator of the differential Renyi's–Tsallis divergence measure.

A Distributed Service-Matching Coverage Via Heterogeneous Agents

We propose a distributed deployment solution for a group of networked agents that should provide a service for a large set of targets, which densely populate a finite area. The agents are heterogeneous in the sense that their quality of service (QoS), modeled as spatial Gaussian distribution, is different. To provide the best service, the objective is to deploy the agents such that their collective QoS distribution is as close as possible to the density distribution of the targets in the sense of the Kullback-Leibler divergence (KLD) measure. We propose a distributed consensus-based expectation-maximization (EM) algorithm to estimate the target density distribution, modeled as a Gaussian mixture model (GMM). Different than the existing algorithms, our proposed distributed EM algorithm enables every agent in the network to obtain an estimate of the GMM model of the distribution of the targets even if only a subset of agents can measure the targets locally. The GMM not only gives an estimate of the targets’ distribution but also clusters the targets to a set of subgroups, each of which is represented by one of the GMM’s Gaussian bases. We use the KLD measure to evaluate the similarity between the QoS distribution of each agent and each Gaussian basis/cluster. A distributed assignment problem is then formulated and solved as a discrete optimal mass transport problem that allocates each agent to a target cluster by taking the KLD as the assignment cost. We demonstrate our results by a sensor deployment for event detection where the sensor’s QoS is modeled as an anisotropic Gaussian distribution.

GPS-Spoofing Attack Detection Technology for UAVs Based on Kullback–Leibler Divergence

Here, we developed a method for detecting cyber security attacks aimed at spoofing the Global Positioning System (GPS) signal of an Unmanned Aerial Vehicle (UAV). Most methods for detecting UAV anomalies indicative of an attack use machine learning or other such methods that compare normal behavior with abnormal behavior. Such approaches require large amounts of data and significant “training” time to prepare and implement the system. Instead, we consider a new approach based on other mathematical methods for detecting UAV anomalies without the need to first collect a large amount of data and describe normal behavior patterns. Doing so can simplify the process of creating an anomaly detection system, which can further facilitate easier implementation of intrusion detection systems in UAVs. This article presents issues related to ensuring the information security of UAVs. Development of the GPS spoofing detection method for UAVs is then described, based on a preliminary study that made it possible to form a mathematical apparatus for solving the problem. We then explain the necessary analysis of parameters and methods of data normalization, and the analysis of the Kullback—Leibler divergence measure needed to detect anomalies in UAV systems.

Empirical underidentification in estimating random utility models: The role of choice sets and standardizations.

A standard approach to distinguishing people’s risk preferences is to estimate a random utility model using a power utility function to characterize the preferences and a logit function to capture choice consistency. We demonstrate that with often‐used choice situations, this model suffers from empirical underidentification, meaning that parameters cannot be estimated precisely. With simulations of estimation accuracy and Kullback–Leibler divergence measures we examined factors that potentially mitigate this problem. First, using a choice set that guarantees a switch in the utility order between two risky gambles in the range of plausible values leads to higher estimation accuracy than randomly created choice sets or the purpose‐built choice sets common in the literature. Second, parameter estimates are regularly correlated, which contributes to empirical underidentification. Examining standardizations of the utility scale, we show that they mitigate this correlation and additionally improve the estimation accuracy for choice consistency. Yet, they can have detrimental effects on the estimation accuracy of risk preference. Finally, we also show how repeated versus distinct choice sets and an increase in observations affect estimation accuracy. Together, these results should help researchers make informed design choices to estimate parameters in the random utility model more precisely.

Dynamic Topology Awareness in Active Distribution Networks Under DG Uncertainties Using GMM-PSEs and KL Divergence

The frequent switching operations and triggering events of automation devices are poorly documented in distribution management systems, leading to challenging work regarding the timely detection of system state changes. Additionally, the limited measurements in distribution networks and the randomness of plug-and-play devices make obtaining real-time observations of system states difficult. Furthermore, distributed generation (DG) output uncertainties exacerbate the situation. Thus, we propose a dynamic topology awareness (DTA) approach to make judgments regarding operational states and identify all possible topology changes (including normal operations, islanding and outages) in the absence of real-time DG measurements. The method consists of two steps. The first step applies parallel Gaussian mixture model-based probabilistic state estimators (GMM-PSEs) of subareas and extended subareas to calculate the probability distributions of border bus voltages with full consideration of DG uncertainties. The second step utilizes the Kullback-Leibler (KL) divergence measure to evaluate the difference between the two probability distributions estimated by subareas and extended subareas, thereby obtaining an indicator (average KL divergence) for identifying possible topologies. The effectiveness of the proposed method is examined on the IEEE 33-bus and IEEE 123-bus test cases. It is computationally efficient and accurate when identifying all possible topologies under limited measurements and DG uncertainties.

Disrupted morphological grey matter networks in early-stage Parkinson\u2019s disease

While previous structural-covariance studies have an advanced understanding of brain alterations in Parkinson's disease (PD), brain–behavior relationships have not been examined at the individual level. This study investigated the topological organization of grey matter (GM) networks, their relation to disease severity, and their potential imaging diagnostic value in PD. Fifty-four early-stage PD patients and 54 healthy controls (HC) underwent structural T1-weighted magnetic resonance imaging. GM networks were constructed by estimating interregional similarity in the distributions of regional GM volume using the Kullback–Leibler divergence measure. Results were analyzed using graph theory and network-based statistics (NBS), and the relationship to disease severity was assessed. Exploratory support vector machine analyses were conducted to discriminate PD patients from HC and different motor subtypes. Compared with HC, GM networks in PD showed a higher clustering coefficient (P = 0.014) and local efficiency (P = 0.014). Locally, nodal centralities in PD were lower in postcentral gyrus and temporal-occipital regions, and higher in right superior frontal gyrus and left putamen. NBS analysis revealed decreased morphological connections in the sensorimotor and default mode networks and increased connections in the salience and frontoparietal networks in PD. Connection matrices and graph-based metrics allowed single-subject classification of PD and HC with significant accuracy of 73.1 and 72.7%, respectively, while graph-based metrics allowed single-subject classification of tremor-dominant and akinetic–rigid motor subtypes with significant accuracy of 67.0%. The topological organization of GM networks was disrupted in early-stage PD in a way that suggests greater segregation of information processing. There is potential for application to early imaging diagnosis.

The extended Kullback-Leibler divergence measure in the unknown probability density function cases and applications

The Kullback-Leibler divergence measure is used to evaluate the similarity between two probability distributions. In theory, the probability density functions are known before applying the formula....

The extended Kullback-Leibler divergence measure in the unknown probability density function cases and applications

The extended Kullback-Leibler divergence measure in the unknown probability density function cases and applications

Seed Investment Bounds for Viral Marketing Under Generalized Diffusion and Selection Guidance

This article attempts to provide viral marketeers guidance in terms of an investment level that could help capture some desired γ percentage of the market share by some target time t with the desired level of confidence. To do this, we first introduce a generalized diffusion model for social networks. A distance-dependent random graph is then considered as a model for the underlying social network, which we use to analyze the proposed diffusion model. Using the fact that vertices degrees have an almost Poisson distribution in distance-dependent random networks, we then provide a lower bound on the probability of the event that the time it takes for an idea (or a product, campaign, disease, and so on) to dominate a prespecified γ percentage of a social network (denoted by R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">γ</sub> ) is smaller than some preselected target time t > 0, i.e., we find a lower bound on the probability of the event {R <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">γ</sub> ≤ t}. Simulation results performed over a wide variety of networks, including random and real world, are then provided to verify that our bound indeed holds in practice. The Kullback-Leibler divergence measure is used to evaluate the performance of our lower bound over these groups of networks, and as expected, we note that for networks that deviate more from the Poisson degree distribution, our lower bound weakens. In the case where absolute/full domination of the market-share is desired, under the linear threshold diffusion model, a particular case of our generalized diffusion model, an upper bound on the size of the seed set is derived, and a selection algorithm is developed to show its tightness. This is also extended to the partial market-share situation.

Inferring geostatistical properties of hydraulic conductivity fields from saline tracer tests and equivalent electrical conductivity time-series

We use Approximate Bayesian Computation and the Kullback–Leibler divergence measure to quantify to what extent horizontal and vertical equivalent electrical conductivity time-series observed during tracer tests constrain the 2-D geostatistical parameters of multivariate Gaussian log-hydraulic conductivity fields. Considering a perfect and known relationship between salinity and electrical conductivity at the point scale, we find that the horizontal equivalent electrical conductivity time-series best constrain the geostatistical properties. The variance, controlling the spreading rate of the solute, is the best constrained geostatistical parameter, followed by the integral scales in the vertical direction. We find that horizontally layered models with moderate to high variance have the best resolved parameters. Since the salinity field at the averaging scale (e.g., the model resolution in tomograms) is typically non-ergodic, our results serve as a starting point for quantifying uncertainty due to small-scale heterogeneity in laboratory-experiments, tomographic results and hydrogeophysical inversions involving DC data.

Protocol-Based Particle Filtering and Divergence Estimation

Network protocols are applied to reduce the amount of data which are transmitted simultaneously over the network. In this article, round-robin protocol (RRP) and try-once-discard protocol (TODP) are studied for state estimation over network where information of different sensor nodes are fused. Using these protocols, at each time sample, only one sensor node can send its information over the network and consequently the required bandwidth is significantly reduced. Particle filters (PF) are able to estimate states of generally any nonlinear and non-Gaussian systems. Therefore, in this article, the development of particle filtering in networked systems under RRP and TODP protocols is studied. Toward this goal, two sequential importance sampling resampling algorithms under RRP and TODP are proposed where their corresponding marginal posterior pdfs are approximated by sequentially computations of weights. For the case of RRP, likelihoods of previous measurements are included in the weight computations while for TODP case, only the latest likelihood appears. Moreover, the approximated marginal posterior pdfs under RRP and TODP are compared with the normal posterior pdf using Kullback-Leibler divergence measure. This measure computes the difference between two probability distributions. Finally, the efficiency of the proposed method is demonstrated for a networked interconnected four-tank system.