Average Kullback-Leibler Divergence Research Articles

Local inconsistency detection using the Kullback–Leibler divergence measure

BackgroundThe standard approach to local inconsistency assessment typically relies on testing the conflict between the direct and indirect evidence in selected treatment comparisons. However, statistical tests for inconsistency have low power and are subject to misinterpreting a p-value above the significance threshold as evidence of consistency.MethodsWe propose a simple framework to interpret local inconsistency based on the average Kullback–Leibler divergence (KLD) from approximating the direct with the corresponding indirect estimate and vice versa. Our framework uses directly the mean and standard error (or posterior mean and standard deviation) of the direct and indirect estimates obtained from a local inconsistency method to calculate the average KLD measure for selected comparisons. The average KLD values are compared with a semi-objective threshold to judge the inconsistency as acceptably low or material. We exemplify our novel interpretation approach using three networks with multiple treatments and multi-arm studies.ResultsAlmost all selected comparisons in the networks were not associated with statistically significant inconsistency at a significance level of 5%. The proposed interpretation framework indicated 14%, 66%, and 75% of the selected comparisons with an acceptably low inconsistency in the corresponding networks. Overall, information loss was more notable when approximating the posterior density of the indirect estimates with that of the direct estimates, attributed to indirect estimates being more imprecise.ConclusionsUsing the concept of information loss between two distributions alongside a semi-objectively defined threshold helped distinguish target comparisons with acceptably low inconsistency from those with material inconsistency when statistical tests for inconsistency were inconclusive.

Dynamic Topology Awareness in Active Distribution Networks Under DG Uncertainties Using GMM-PSEs and KL Divergence

The frequent switching operations and triggering events of automation devices are poorly documented in distribution management systems, leading to challenging work regarding the timely detection of system state changes. Additionally, the limited measurements in distribution networks and the randomness of plug-and-play devices make obtaining real-time observations of system states difficult. Furthermore, distributed generation (DG) output uncertainties exacerbate the situation. Thus, we propose a dynamic topology awareness (DTA) approach to make judgments regarding operational states and identify all possible topology changes (including normal operations, islanding and outages) in the absence of real-time DG measurements. The method consists of two steps. The first step applies parallel Gaussian mixture model-based probabilistic state estimators (GMM-PSEs) of subareas and extended subareas to calculate the probability distributions of border bus voltages with full consideration of DG uncertainties. The second step utilizes the Kullback-Leibler (KL) divergence measure to evaluate the difference between the two probability distributions estimated by subareas and extended subareas, thereby obtaining an indicator (average KL divergence) for identifying possible topologies. The effectiveness of the proposed method is examined on the IEEE 33-bus and IEEE 123-bus test cases. It is computationally efficient and accurate when identifying all possible topologies under limited measurements and DG uncertainties.

Deep Neural Network Classification of Tactile Materials Explored by Tactile Sensor Array With Various Active-Cell Formations

Reducing the input data of tactile sensory systems brings a large degree of freedom to real-world implementations from the perspectives of bandwidth and computational complexity. For this, in this letter, we suggest efficient active-cell formations with a high classification accuracy of tactile materials. By revealing that averaged Kullback–Leibler-divergence and common frequency component power to variance ratio are proportional to the classification accuracy, we showed that those methods can be useful in estimating valid active-cell formations.

Distributed Joint Sensor Registration and Multitarget Tracking via Sensor Network

This paper addresses distributed registration of a sensor network for multitarget tracking. Each sensor gets measurements of the target position in a local coordinate frame, having no knowledge about the relative positions (referred to as drift parameters) and azimuths (referred to as orientation parameters) of its neighboring nodes. The multitarget set is modeled as an independent and identically distributed (i.i.d.) cluster random finite set (RFS), and a consensus cardinality probability hypothesis density (CPHD) filter is run over the network to recursively compute in each node the posterior RFS density. Then a suitable cost function, xpressing the discrepancy between the local posteriors in terms of averaged Kullback-Leibler divergence, is minimized with respect to the drift and orientation parameters for sensor registration purposes. In this way, a computationally feasible optimization approach for joint sensor registraton and multitarget tracking is devised. Finally, the effectiveness of the proposed approach is demonstrated through simulation experiments on both tree networks and networks with cycles, as well as with both linear and nonlinear sensors.

DETECTING NONLINEAR AND NONSTATIONARY PROPERTIES OF POST-APNEIC SNORING SOUNDS USING HILBERT–HUANG TRANSFORM

This study focuses on patients with severe obstructive sleep apnea syndrome (OSAS), and clarifies the existence of nonlinear and nonstationary properties in post-apneic snoring sounds. Many researchers have tried to discover intrinsic properties of the snoring sounds in OSAS patients for the past decades using linear frequency analysis, but no one has shown any evidence of the existence of nonlinearity and nonstationarity based on the quantitative evaluation of the post-apneic snoring sounds. In this study, Hilbert–Huang transform (HHT), which is designed for analyzing nonlinear and nonstationary signals, is adopted to generate a time-frequency map and the temporal variation in the spectral density is quantitatively evaluated using the averaged Kullback–Leibler divergence (AKL). As a result, for six OSAS patients, there is a tendency that most of AKL calculated from post-apneic snores are higher than those from non-apneic snores, which indicates that post-apneic snores are more nonstationary. In addition, we also evaluated the difference between the HHT time-frequency maps and spectrograms using short-time fourier transforms (STFT). Such analyses revealed that frequency fluctuations inherent to snoring can be adequately represented through HHT, but not with STFT. These nonlinear and nonstationary properties seem to be highly related to the physiological phenomena of OSAS, two of which are the explosive airflow after reopening of the closed airway and the collision vibration of the soft tissues.

Distributed joint sensor registration and target tracking via sensor network

This paper deals with distributed registration of a sensor network for target tracking in the presence of false and/or missed measurements. Each sensor acquires measurements of the target position in local coordinates, having no knowledge about the relative positions (referred to as drift parameters) of its neighboring nodes. A distributed Bernoulli filter is run over the network to compute in each node a local posterior target density. Then a suitable cost function, expressing the discrepancy between the local posteriors in terms of averaged Kullback–Leibler divergence, is minimized with respect to the drift parameters for sensor registration purposes. In this way, a computationally feasible optimization approach for joint sensor registration and target tracking is devised. Finally, the effectiveness of the proposed approach is demonstrated through simulation experiments on both tree networks and networks with cycles, as well as with both linear and nonlinear sensors.

Using Bisect K-Means Clustering Technique in the Analysis of Arabic Documents

In this article, I have investigated the performance of the bisect K-means clustering algorithm compared to the standard K-means algorithm in the analysis of Arabic documents. The experiments included five commonly used similarity and distance functions (Pearson correlation coefficient, cosine, Jaccard coefficient, Euclidean distance, and averaged Kullback-Leibler divergence) and three leading stemmers. Using the purity measure, the bisect K-means clearly outperformed the standard K-means in all settings with varying margins. For the bisect K-means, the best purity reached 0.927 when using the Pearson correlation coefficient function, while for the standard K-means, the best purity reached 0.884 when using the Jaccard coefficient function. Removing stop words significantly improved the results of the bisect K-means but produced minor improvements in the results of the standard K-means. Stemming provided additional minor improvement in all settings except the combination of the averaged Kullback-Leibler divergence function and the root-based stemmer, where the purity was deteriorated by more than 10%. These experiments were conducted using a dataset with nine categories, each of which contains 300 documents.

Emergence of scale-free characteristics in socio-ecological systems with bounded rationality.

Socio–ecological systems are increasingly modelled by games played on complex networks. While the concept of Nash equilibrium assumes perfect rationality, in reality players display heterogeneous bounded rationality. Here we present a topological model of bounded rationality in socio-ecological systems, using the rationality parameter of the Quantal Response Equilibrium. We argue that system rationality could be measured by the average Kullback–-Leibler divergence between Nash and Quantal Response Equilibria, and that the convergence towards Nash equilibria on average corresponds to increased system rationality. Using this model, we show that when a randomly connected socio-ecological system is topologically optimised to converge towards Nash equilibria, scale-free and small world features emerge. Therefore, optimising system rationality is an evolutionary reason for the emergence of scale-free and small-world features in socio-ecological systems. Further, we show that in games where multiple equilibria are possible, the correlation between the scale-freeness of the system and the fraction of links with multiple equilibria goes through a rapid transition when the average system rationality increases. Our results explain the influence of the topological structure of socio–ecological systems in shaping their collective cognitive behaviour, and provide an explanation for the prevalence of scale-free and small-world characteristics in such systems.

Dose impact in radiographic lung injury following lung SBRT: Statistical analysis and geometric interpretation

To demonstrate a new method of evaluating dose response of treatment-induced lung radiographic injury post-SBRT (stereotactic body radiotherapy) treatment and the discovery of bimodal dose behavior within clinically identified injury volumes. Follow-up CT scans at 3, 6, and 12 months were acquired from 24 patients treated with SBRT for stage-1 primary lung cancers or oligometastic lesions. Injury regions in these scans were propagated to the planning CT coordinates by performing deformable registration of the follow-ups to the planning CTs. A bimodal behavior was repeatedly observed from the probability distribution for dose values within the deformed injury regions. Based on a mixture-Gaussian assumption, an Expectation-Maximization (EM) algorithm was used to obtain characteristic parameters for such distribution. Geometric analysis was performed to interpret such parameters and infer the critical dose level that is potentially inductive of post-SBRT lung injury. The Gaussian mixture obtained from the EM algorithm closely approximates the empirical dose histogram within the injury volume with good consistency. The average Kullback-Leibler divergence values between the empirical differential dose volume histogram and the EM-obtained Gaussian mixture distribution were calculated to be 0.069, 0.063, and 0.092 for the 3, 6, and 12 month follow-up groups, respectively. The lower Gaussian component was located at approximately 70% prescription dose (35 Gy) for all three follow-up time points. The higher Gaussian component, contributed by the dose received by planning target volume, was located at around 107% of the prescription dose. Geometrical analysis suggests the mean of the lower Gaussian component, located at 35 Gy, as a possible indicator for a critical dose that induces lung injury after SBRT. An innovative and improved method for analyzing the correspondence between lung radiographic injury and SBRT treatment dose has been demonstrated. Bimodal behavior was observed in the dose distribution of lung injury after SBRT. Novel statistical and geometrical analysis has shown that the systematically quantified low-dose peak at approximately 35 Gy, or 70% prescription dose, is a good indication of a critical dose for injury. The determined critical dose of 35 Gy resembles the critical dose volume limit of 30 Gy for ipsilateral bronchus in RTOG 0618 and results from previous studies. The authors seek to further extend this improved analysis method to a larger cohort to better understand the interpatient variation in radiographic lung injury dose response post-SBRT.

Arabic Text Summarization Based on Latent Semantic Analysis to Enhance Arabic Documents Clustering

Arabic Documents Clustering is an important task for obtaining good results with the traditional Information Retrieval (IR) systems especially with the rapid growth of the number of online documents present in Arabic language. Documents clustering aim to automatically group similar documents in one cluster using different similarity/distance measures. This task is often affected by the documents length, useful information on the documents is often accompanied by a large amount of noise, and therefore it is necessary to eliminate this noise while keeping useful information to boost the performance of Documents clustering. In this paper, we propose to evaluate the impact of text summarization using the Latent Semantic Analysis Model on Arabic Documents Clustering in order to solve problems cited above, using five similarity/distance measures: Euclidean Distance, Cosine Similarity, Jaccard Coefficient, Pearson Correlation Coefficient and Averaged Kullback-Leibler Divergence, for two times: without and with stemming. Our experimental results indicate that our proposed approach effectively solves the problems of noisy information and documents length, and thus significantly improve the clustering performance.

Asymptotically minimax Bayesian predictive densities for multinomial models

One-step ahead prediction for the multinomial model is considered. The performance of a predictive density is evaluated by the average Kullback-Leibler divergence from the true density to the predictive density. Asymptotic approximations of risk functions of Bayesian predictive densities based on Dirichlet priors are obtained. It is shown that a Bayesian predictive density based on a specific Dirichlet prior is asymptotically minimax. The asymptotically minimax prior is different from known objective priors such as the Jeffreys prior or the uniform prior.

Grammatical error simulation for computer-assisted language learning

This paper presents an automated method to generate realistic grammatical errors that can perform crucial functions for advanced technologies in computer-assisted language learning (CALL), including generating corrective feedback in dialog-based CALL (DB-CALL) systems, simulating a language learner to optimize tutoring strategies, and generating context-dependent grammar quizzes as educational materials. The goal of this study is to make grammatical errors generated by automatic simulators more realistic. To generate realistic errors, expert knowledge of language learners’ error characteristics was imported into a statistical modeling system that uses Markov logic, which provides a theoretically sound way to encode knowledge into probabilistic first-order logic. We learned the weights of first-order formulas from a learner corpus. The improved quality of the proposed method was demonstrated through comparative experiments using automatic evaluations (precision and recall rate and Kullback–Leibler divergence between error distributions) and human assessments. The proposed method increased precision by 6% and recall by 8.33% averaged across all proficiency levels. It also exhibited a relative improvement of 37.5% in the average Kullback–Leibler divergence. Judgment by human evaluators showed that the proposed method increased the average scores in two different evaluation tasks by 7 and by 0.411. Finally, we present a measure of labor savings to help predict the time and cost associated with this method for those who plan to exploit grammatical error simulation for their applications. The results indicate that using the proposed method could reduce the grammatical error generation time by 59% in average.

An Estimating Method for Parametric Spectral Densities of Gaussian Time Series

An estimating method for spectral densities of Gaussian time series that belong to a parametric model is proposed. Spectral density estimators are evaluated by using average Kullback–Leibler divergence from the true spectral density to estimated spectral densities. In the classical approach, unknown spectral densities are estimated by replacing the unknown parameters by asymptotically efficient estimates. In the alternative method introduced in the present paper, spectral density estimates usually do not belong to the model. The alternative spectral density estimators asymptotically dominate the classical ones. The difference in average Kullback–Leibler divergence between them can be regarded as the mixture mean curvature of the model in the space of all spectral densities. The explicit expression for the proposed estimators of spectral densities of autoregressive processes is obtained. The accuracy of prediction can be improved by using predictors that correspond to the alternative spectral density estimators.

On asymptotic properties of predictive distributions

SUMMARY Second-order asymptotic theory of predictive distributions is investigated. It is shown that estimative distributions with asymptotically efficient estimators can be improved to predictive distributions that do not belong to the model. The model is assumed to be a multidimensional curved exponential family. Predictive distributions can be constructed by shifting estimative distributions in a direction orthogonal to the model. The average Kullback-Leibler divergence from the true distribution to a predictive distribution is represented as the sum of two components. One of them depends on the choice of the estimative distribution and the other depends on the shift orthogonal to the model. The optimal shift orthogonal to the model is obtained. A quantity that can be regarded as the mixture mean curvature of the model in the space of all probability distributions is introduced. The difference in average Kullback-Leibler divergence between an estimative distribution and the optimal predictive distribution obtained by shifting it can be regarded as the mixture mean curvature of the model. An asymptotic expression for Bayesian predictive distributions is also obtained.

Robust predictive distributions for exponential families

SUMMARY The bootstrap predictive distribution considered by Harris (1989) is modified by replacing the maximum likelihood estimator of the unknown parameter by its minimum Hellinger distance estimator. The predictive distribution thus obtained is asymptotically superior to the corresponding estimative distribution in terms of the average Kullback-Leibler divergence, when the true distribution is in the natural exponential family. Simulation results are provided for the binomial and normal distributions which suggest that the proposed predictive distributions are robust and can perform better than the likelihood based methods under data contamination.

Predictive fit for natural exponential families

SUMMARY The paper examines predictive distributions, concentrating on measuring their fit to the true distribution by average Kullback-Leibler divergence. The notion of an 'averaged bootstrap' predictive distribution is introduced. This predictive distribution is shown to be asymptotically superior to the estimative distribution, in terms of average KullbackLeibler divergence, when the true distribution is in a natural exponential family. Smallsample results are presented for the Poisson and binomial distributions which suggest that the bootstrap distribution performs well in these cases.