Kullback-Leibler Divergence Measure Research Articles

Efficacy of Texture, Shape, and Intensity Feature Fusion for Posterior-Fossa Tumor Segmentation in MRI

Our previous works suggest that fractal texture feature is useful to detect pediatric brain tumor in multimodal MRI. In this study, we systematically investigate efficacy of using several different image features such as intensity, fractal texture, and level-set shape in segmentation of posterior-fossa (PF) tumor for pediatric patients. We explore effectiveness of using four different feature selection and three different segmentation techniques, respectively, to discriminate tumor regions from normal tissue in multimodal brain MRI. We further study the selective fusion of these features for improved PF tumor segmentation. Our result suggests that Kullback-Leibler divergence measure for feature ranking and selection and the expectation maximization algorithm for feature fusion and tumor segmentation offer the best results for the patient data in this study. We show that for T1 and fluid attenuation inversion recovery (FLAIR) MRI modalities, the best PF tumor segmentation is obtained using the texture feature such as multifractional Brownian motion (mBm) while that for T2 MRI is obtained by fusing level-set shape with intensity features. In multimodality fused MRI (T1, T2, and FLAIR), mBm feature offers the best PF tumor segmentation performance. We use different similarity metrics to evaluate quality and robustness of these selected features for PF tumor segmentation in MRI for ten pediatric patients.

A New Distance Measure for a Variable-Sized Acoustic Model Based on MDL Technique

Embedding a large vocabulary speech recognition system in mobile devices requires a reduced acoustic model obtained by eliminating redundant model parameters. In conventional optimization methods based on the minimum description length (MDL) criterion, a binary Gaussian tree is built at each state of a hidden Markov model by iteratively finding and merging similar mixture components. An optimal subset of the tree nodes is then selected to generate a downsized acoustic model. To obtain a better binary Gaussian tree by improving the process of finding the most similar Gaussian components, this paper proposes a new distance measure that exploits the difference in likelihood values for cases before and after two components are combined. The mixture weight of Gaussian components is also introduced in the component merging step. Experimental results show that the proposed method outperforms MDL-based optimization using either a Kullback-Leibler (KL) divergence or weighted KL divergence measure. The proposed method could also reduce the acoustic model size by 50% with less than a 1.5% increase in error rate compared to a baseline system.

Development of theoretical approaches to the control of a water-treatment system

The Advanced Life Support/NASA Specialized Center of Research and Training (ALS/NSCORT) focuses on research and development of technologies to support human habitation during space missions. This research was done as part of an effort to maintain crewmembers’ water supply in a closed life-support system. The water subsystem was the primary focus of this study because water is one of the most expensive and important resources for human survival. This paper focuses on the application of previously-developed methodologies for controlling a water-treatment system. The water-treatment and usage model was set up based on the Baseline Value and Assumptions Document (BVAD) published by NASA. The conditions, or “states”, of the system are captured hourly in the model. Control policies are defined in terms of treatment efficiencies. One control policy is to be used for the system at the beginning of each hour. A simulation is developed to study the impact of different control policies to this model. The Markov Decision Process (MDP) is used to aid the decision of choosing a policy to be used. In order for the MDP to pick the best policy, a state-space exploration technique, the Controlled Random Walk, is used for state-space exploration. This technique, together with re-initialization of initial system conditions, allow the system to visit a larger portion of the state space compared to some other techniques. In order to describe the state transitions, transition-probability matrices need to be computed. A probability-convergence technique is developed to minimize bias while computing these matrices. This technique is based on the Kullback–Leibler divergence measure, but modified for this application. A test case is constructed to assess the performance of best policies. Three scenarios with different levels of fluctuation, or disturbances, of water-consumption and treatment efficiencies are introduced in the test case. Results show that the best policies outperformed cases where fixed policies, as well as random policies, are used in all three scenarios.

Relative Sensitivity of Conditional Distributions to Kurtosis Deviations in the Joint Model

The Multivariate Exponential Power family is considered for n-dimensional random variables, Z, with a known partition Z (Y,X) of dimensions p and n-p, respectively. An infinitesimal variation of any parameter produces both conditional and marginal distributions perturbations. The aim of our study is to determine the local effect of kurtosis deviations by means of the Kullback-Leibler divergence measure between probability distributions. The additive decomposition of this measure in terms of the conditional and marginal distributions, YǀX and X, has been used to define the relative sensitivity of the conditional distributions family {YǀX=x}. The obtained results show that, for large dimensions, it is nearly p/n.

Emergence of consensus and shared vocabularies in collaborative tagging systems

This article uses data from the social bookmarking site del.icio.us to empirically examine the dynamics of collaborative tagging systems and to study how coherent categorization schemes emerge from unsupervised tagging by individual users. First, we study the formation of stable distributions in tagging systems, seen as an implicit form of “consensus” reached by the users of the system around the tags that best describe a resource. We show that final tag frequencies for most resources converge to power law distributions and we propose an empirical method to examine the dynamics of the convergence process, based on the Kullback-Leibler divergence measure. The convergence analysis is performed for both the most utilized tags at the top of tag distributions and the so-called long tail. Second, we study the information structures that emerge from collaborative tagging, namely tag correlation (or folksonomy) graphs. We show how community-based network techniques can be used to extract simple tag vocabularies from the tag correlation graphs by partitioning them into subsets of related tags. Furthermore, we also show, for a specialized domain, that shared vocabularies produced by collaborative tagging are richer than the vocabularies which can be extracted from large-scale query logs provided by a major search engine. Although the empirical analysis presented in this article is based on a set of tagging data obtained from del.icio.us, the methods developed are general, and the conclusions should be applicable across other websites that employ tagging.

A context tree method for multistage fault detection and isolation with applications to commercial video broadcasting systems

Many systems have distributed functionalities among several autonomous components. Such complex systems can generally be described by finite state machines whose behavior is often non-linear and context-dependent. This paper proposes a generic system model based on context trees to predict system behavior for the purpose of fault detection and isolation in a multistage serial system. The approach starts with learning multistage model structures by capturing the expected statistical distribution of the input/output at different stages. The estimated model is then employed to detect departures by comparing the contexts of the new system output with a set of optimal contexts for each stage using the Kullback–Leibler divergence measure. Problems can then be isolated to the stage that contributes the most to these differences. The methodology is demonstrated by an application in commercial video broadcasting systems.

Predictive ability of models for calving difficulty in US Holsteins

The performance of alternative threshold models for analyzing calving difficulty (CD) in Holstein cows was evaluated in terms of predictive ability. Four models were considered, with CD classified into either three or four categories and analysed either as a single trait or jointly with gestation length (GL). The data contained GL and CD records from 90 393 primiparous cows, sired by 1122 bulls and distributed over 935 herd-calving year classes. Predictive ability of each model was evaluated using four criteria: mean squared error of the difference between observed and predicted CD scores; a Kullback-Leibler divergence measure between the observed and predicted distributions of CD scores; Pearson's correlation between observed and predicted CD scores and ability to correctly classify bulls as above or below average for incidence of CD. In general, the four models had similar predictive abilities. The joint analysis of CD with GL produced little, if any, improvement in predictive ability over univariate models. In light of the small difference in predictive ability between models treating CD with three or four categories and considering that a greater number of categories can provide more information, analysis of CD classified into four categories seems warranted.

A Kullback-Leibler View of Linear and Log-Linear Pools

Linear and log-linear pools are widely used methods for aggregating expert belief. This paper frames the expert aggregation problem as a decision problem with scoring rules. We propose a scoring function that uses the Kullback-Leibler (KL) divergence measure between the aggregate distribution and each of the expert distributions. The asymmetric nature of the KL measure allows for a convenient scoring system for which the linear and log-linear pools provide the optimal assignment. We also propose a “goodness-of-fit” measure that determines how well each opinion pool characterizes its expert distributions, and also determines the performance of each pool under this scoring function. We work through several examples to illustrate the approach.

Analyzing the effect of introducing a kurtosis parameter in Gaussian Bayesian networks

Gaussian Bayesian networks are graphical models that represent the dependence structure of a multivariate normal random variable with a directed acyclic graph (DAG). In Gaussian Bayesian networks the output is usually the conditional distribution of some unknown variables of interest given a set of evidential nodes whose values are known. The problem of uncertainty about the assumption of normality is very common in applications. Thus a sensitivity analysis of the non-normality effect in our conclusions could be necessary. The aspect of non-normality to be considered is the tail behavior. In this line, the multivariate exponential power distribution is a family depending on a kurtosis parameter that goes from a leptokurtic to a platykurtic distribution with the normal as a mesokurtic distribution. Therefore a more general model can be considered using the multivariate exponential power distribution to describe the joint distribution of a Bayesian network, with a kurtosis parameter reflecting deviations from the normal distribution. The sensitivity of the conclusions to this perturbation is analyzed using the Kullback–Leibler divergence measure that provides an interesting formula to evaluate the effect.

A Bayesian Hierarchical Model for Analysis of Single-Nucleotide Polymorphisms Diversity in Multilocus, Multipopulation Samples

The distribution of genetic variation among populations is conveniently measured by Wright's FST, which is a scaled variance taking on values in [0,1]. For certain types of genetic markers and for single-nucleotide polymorphisms (SNPs) in particular, it is reasonable to presume that allelic differences at most loci are selectively neutral. For such loci, the distribution of genetic variation among populations is determined by the size of local populations, the pattern and rate of migration among those populations, and the rate of mutation. Because the demographic parameters (population sizes and migration rates) are common across all autosomal loci, locus-specific estimates of FST will depart from a common distribution only for loci with unusually high or low rates of mutation or for loci that are closely associated with genomic regions having a relationship with fitness. Thus, loci that are statistical outliers showing significantly more among-population differentiation than others may mark genomic regions subject to diversifying selection among the sample populations. Similarly, statistical outliers showing significantly less differentiation among populations than others may mark genomic regions subject to stabilizing selection across the sample populations. We propose several Bayesian hierarchical models to estimate locus-specific effects on FST, and we apply these models to single nucleotide polymorphism data from the HapMap project. Because loci that are physically associated with one another are likely to show similar patterns of variation, we introduce conditional autoregressive models to incorporate the local correlation among loci for high-resolution genomic data. We estimate the posterior distributions of model parameters using Markov chain Monte Carlo (MCMC) simulations. Model comparison using several criteria, including deviance information criterion (DIC) and pseudomarginal likelihood (LPML), reveals that a model with locus- and population-specific effects is superior to other models for the data used in the analysis. To detect statistical outliers we propose an approach that measures divergence between the posterior distributions of locus-specific effects and the common FST with the Kullback-Leibler divergence measure. We calibrate this measure by comparing values with those produced from the divergence between a biased and a fair coin. We conduct a simulation study to illustrate the performance of our approach for detecting loci subject to stabilizing/divergent selection, and we apply the proposed models to low- and high-resolution SNP data from the HapMap project. Model comparison using DIC and LPML reveals that conditional autoregressive (CAR) models are superior to alternative models for the high-resolution data. For both low- and high- resolution data, we identify statistical outliers that are associated with known genes.

Target Tracking Using a Joint Acoustic Video System

In this paper, a multitarget tracking system for collocated video and acoustic sensors is presented. We formulate the tracking problem using a particle filter based on a state-space approach. We first discuss the acoustic state-space formulation whose observations use a sliding window of direction-of-arrival estimates. We then present the video state space that tracks a target's position on the image plane based on online adaptive appearance models. For the joint operation of the filter, we combine the state vectors of the individual modalities and also introduce a time-delay variable to handle the acoustic-video data synchronization issue, caused by acoustic propagation delays. A novel particle filter proposal strategy for joint state-space tracking is introduced, which places the random support of the joint filter where the final posterior is likely to lie. By using the Kullback-Leibler divergence measure, it is shown that the joint operation of the filter decreases the worst case divergence of the individual modalities. The resulting joint tracking filter is quite robust against video and acoustic occlusions due to our proposal strategy. Computer simulations are presented with synthetic and field data to demonstrate the filter's performance

Synthesis of image deformation strategies

Warping is one of the key areas of image analysis but there has been no understanding of the effects of different non-linear deformations in literature. This paper addresses the problem of the distortion effect produced by different types of non-linear deformation strategies on textured images. The images are modelled by a Gaussian random field. We first give various examples to illustrate that the model generates realistic images. We consider two types of deformations—a deterministic deformation and a landmark based deformation. The latter includes various radial basis type deformations including the thin-plate splines based deformation. The effects of deformations are assessed through Kullback–Leibler divergence measure. The measure is estimated by statistical sampling techniques. It is found empirically that this divergence measure is approximately distributed as a lognormal distribution under various different deformations. Thus a coefficient of variation based on log-divergence provides a natural criterion to compare different types of deformations. It is found that the thin-plate splines deformation is almost optimal over the wider class of the radial type deformations.

Bayesian nonparametric model selection and model testing

This article examines a Bayesian nonparametric approach to model selection and model testing, which is based on concepts from Bayesian decision theory and information theory. The approach can be used to evaluate the predictive-utility of any model that is either probabilistic or deterministic, with that model analyzed under either the Bayesian or classical-frequentist approach to statistical inference. Conditional on an observed set of data, generated from some unknown true sampling density, the approach identifies the “best” model as the one that predicts a sampling density that explains the most information about the true density. Furthermore, in the approach, the decision is to reject a model when it does not explain enough information about the true density (according to a straightforward calibration of the Kullback–Leibler divergence measure). The posterior estimate of the true density is based on a Bayesian nonparametric prior that can give positive support to the entire space of sampling densities (defined on some sample space). This article also discusses the theoretical and practical advantages of the Bayesian nonparametric approach over all other types of model selection procedures, and over any model testing procedure that depends on interpreting a p -value. Finally, the Bayesian nonparametric approach is illustrated on four real data sets, in the comparison and testing of order-constrained models, cognitive models, models of choice-behavior, and a test of a general psychometric model.

Universality of probability density distributions in the overlap region in high Reynolds number turbulent boundary layers

The probability density functions (PDFs) of the streamwise mean velocity in high Reynolds number turbulent boundary layers have been studied. The hypothesis of self-similar, normalized with the root mean square velocity, PDFs has been tested using the KTH database of high Reynolds number zero pressure-gradient turbulent boundary layer flow. The self-similarity was tested using the Kullback–Leibler divergence measure and it was found that the region of self-similar PDFs extends from about 160 viscous wall units to about 0.3 boundary layer thicknesses (in δ95). This region is somewhat larger than the universal overlap region. The PDF shape in the universal overlap region is close to Gaussian allowing for a Gram–Charlier expansion approximation of the measured PDFs. A remarkable collapse was found for 57 normalized PDF distributions for different positions within the universal overlap region and Reynolds numbers based on the momentum-loss thickness between 4300 and 12 600, strongly indicating a high degree of flow universality within the universal overlap region. Within the range studied, the Gram–Charlier expansion coefficients (related to the PDF moments) show no Reynolds number trend further supporting the self-similarity hypothesis.

A New Kullback–Leibler VAD for Speech Recognition in Noise

This letter shows an innovative voice activity detector (VAD) based on the Kullback-Leibler (KL) divergence measure. The algorithm is evaluated in the context of the recently approved ETSI standard for distributed speech recognition (DSR). The VAD uses long-term information of the noisy speech signal in order to define a more robust decision rule yielding high accuracy. The mel-scaled filter bank log-energies (FBE) are modeled by means of Gaussian distributions, and a symmetric KL divergence is used for the estimation of the distance between speech and noise distributions. The decision rule is formulated in terms of the average subband KL divergence that is compared to a noise-adaptable threshold. An exhaustive analysis using the AURORA databases is conducted in order to assess the performance of the proposed method and to compare it to existing standard VAD methods.

Noninformative priors, credible sets and Bayesian hypothesis testing for the intraclass model

The objective of this paper is twofold. The first is to find a Bayesian credible interval for the intraclass correlation coefficient in symmetric normal models based on some “default” or “noninformative” priors. Probability matching priors and reference priors along with Jeffreys’ prior are considered, and the one-at-a-time reference prior emerges as the “optimal” according to several criteria. The second objective is to compare two nested models such as the intraclass and independence models using the distance or divergence between the two as the basis of comparison. A suitable criterion for this is the “power divergence measure” as introduced by Cressie and Read (J. Roy. Statist. Soc. Ser. B 46 (1984) 440). Such a measure includes the two Kullback-Leibler divergence measures and the Hellinger divergence measure as special cases. For the specific problem, the power divergence measure turns out to be a function solely of ρ, the intraclass correlation coefficient. Also, this function is increasing for ρ>0, is decreasing for ρ<0, and the minimum is attained at ρ=0. Thus the model comparison problem in this case amounts to testing the hypothesis H 0 : ρ=0 . Due to the duality between hypothesis tests and set estimation, the hypothesis testing problem can also be solved by solving a corresponding set estimation problem. The present paper develops Bayesian methods based on the Kullback-Leibler and Hellinger divergence measures, rejecting H 0 : ρ=0 when the specified divergence measure exceeds some number d. This number d is so chosen that the resulting credible interval for the divergence measure has specified coverage probability 1− α. The length of such an interval is compared with the (i) equal two-tailed credible interval and (ii) the HPD credible interval for ρ with the same coverage probability which can also be inverted into acceptance regions of H 0 : ρ=0 . An example is considered where the HPD interval based on the one-at-a-time reference prior turns out to be the shortest credible interval having the same coverage probability.

Universality in the brain while listening to music.

The human brain, which is one of the most complex organic systems, involves billions of interacting physiological and chemical processes that give rise to experimentally observed neuroelectrical activity, which is called an electroencephalogram (EEG). The presence of non-stationarity and intermittency render standard available methods unsuitable for detecting hidden dynamical patterns in the EEG. In this paper, a method that is suitable for non-stationary signals and preserving the phase characteristics and that combines wavelet and Hilbert transforms was applied to multivariate EEG signals from human subjects at rest as well as in different cognitive states: listening to music, listening to text and performing spatial imagination. It was found that, if suitably rescaled, the gamma band EEG over distributed brain areas while listening to music can be described by a universal and homogeneous scaling, whereas this homogeneity in scale is reduced at resting conditions and also during listening to text and performing spatial imagination. The degree of universality is characterized by a Kullback-Leibler divergence measure. By statistical surrogate analysis, nonlinear phase interaction was found to play an important role in exhibiting universality among multiple cortical regions.

An alternative bayesian approach to the multivariate behrens-fisher problem

In this note, an alternative approach to the multivariate Behrens-Fisher problem is presented in the Bayesian setup. We use the Kullback-Leibler di-vergence measure to assess the discrepancy between the “null” model and the “alternative” model. We then use the Bayesian approach to minimize the posterior expectation of the distance between the two models. A simulation study using bootstrap methods in the bivariate case is presented to demonstrate the implementation of the test procedure derived.

On a unification of divergences by means of information amounts

The notion of discriminating amount associated with the notion of a divergence is given. It is based on the concept of Rényi's order α relative entropy and Burbea-Rao's K α -divergence measure. Moreover, the K α -divergence measure and some order α Kullback-Leibler's divergence measures are unified by means of (α,β) discriminating amounts. Local properties of these amounts are studied on the statistical parameter space.

Local divergence and association

SUMMARY In applications of differential geometry to problems of parametric statistical inference, the notion of divergence is often used to measure the separation between two parametric densities. In this paper we move away from the parametric framework of much of statistical differential geometry and instead employ divergence in a covariate setting. Many data-analyses involve investigation of how the conditional distribution f(y I X = x) of a random variable Y changes with covariate values x. We propose using local divergence between conditional distributions as a measure of this change and thus as a general measure of association between Y and the covariates X. Under certain regularity conditions we define a class of divergences which are locally the Rao distance (LR). The limiting LR divergence is bounded below by the signal-to-noise ratio, with equality holding if and only if the conditional density comes from the natural exponential family. The correlation curve and in particular the correlation coefficient are simply transformations of the signal-to-noise ratio and thus are transformations of local divergence. We therefore obtain a differential geometric interpretation of standard ideas of association. The class of LR divergences is broad and includes the Kullback-Leibler divergence and Renyi oc-information measures. We can therefore interpret local association as local utility or information gain of order cx. We obtain comparable results without regularity conditions.