Statistical Divergences Research Articles

Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics.

Statistical divergences are important tools in data analysis, information theory, and statistical physics, and there exist well-known inequalities on their bounds. However, in many circumstances involving temporal evolution, one needs limitations on the rates of such quantities instead. Here, several general upper bounds on the rates of some f-divergences are derived, valid for any type of stochastic dynamics (both Markovian and non-Markovian), in terms of information-like and/or thermodynamic observables. As special cases, the analytical bounds on the rate of mutual information are obtained. The major role in all those limitations is played by temporal Fisher information, characterizing the speed of global system dynamics, and some of them contain entropy production, suggesting a link with stochastic thermodynamics. Indeed, the derived inequalities can be used for estimation of minimal dissipation and global speed in thermodynamic stochastic systems. Specific applications of these inequalities in physics and neuroscience are given, which include the bounds on the rates of free energy and work in nonequilibrium systems, limits on the speed of information gain in learning synapses, as well as the bounds on the speed of predictive inference and learning rate. Overall, the derived bounds can be applied to any complex network of interacting elements, where predictability and thermodynamics of network dynamics are of prime concern.

Bayesian estimation of the Kullback-Leibler divergence for categorical systems using mixtures of Dirichlet priors.

In many applications in biology, engineering, and economics, identifying similarities and differences between distributions of data from complex processes requires comparing finite categorical samples of discrete counts. Statistical divergences quantify the difference between two distributions. However, their estimation is very difficult and empirical methods often fail, especially when the samples are small. We develop a Bayesian estimator of the Kullback-Leibler divergence between two probability distributions that makes use of a mixture of Dirichlet priors on the distributions being compared. We study the properties of the estimator on two examples: probabilities drawn from Dirichlet distributions and random strings of letters drawn from Markov chains. We extend the approach to the squared Hellinger divergence. Both estimators outperform other estimation techniques, with better results for data with a large number of categories and for higher values of divergences.

A two-parameter family of non-parametric, deformed exponential manifolds

We construct a new family of non-parametric statistical manifolds by means of a two-parameter class of deformed exponential functions, that includes functions with power-law, linear and sublinear rates of growth. The manifolds are modelled on weighted, mixed-norm Sobolev spaces that are especially suited to this purpose, in the sense that an important class of nonlinear superposition operators (those used in the construction of divergences and tensors) act continuously on them. We analyse variants of these operators, that map into “subordinate” Sobolev spaces, and evaluate the associated gain in regularity. With appropriate choice of parameter values, the manifolds support a large variety of the statistical divergences and entropies appearing in the literature, as well as their associated tensors, eg. the Fisher-Rao metric. Manifolds of finite measures and probability measures are constructed; the latter are shown to be smoothly embedded submanifolds of the former.

Local Intrinsic Dimensionality, Entropy and Statistical Divergences.

Properties of data distributions can be assessed at both global and local scales. At a highly localized scale, a fundamental measure is the local intrinsic dimensionality (LID), which assesses growth rates of the cumulative distribution function within a restricted neighborhood and characterizes properties of the geometry of a local neighborhood. In this paper, we explore the connection of LID to other well known measures for complexity assessment and comparison, namely, entropy and statistical distances or divergences. In an asymptotic context, we develop analytical new expressions for these quantities in terms of LID. This reveals the fundamental nature of LID as a building block for characterizing and comparing data distributions, opening the door to new methods for distributional analysis at a local scale.

Statistical Divergences between Densities of Truncated Exponential Families with Nested Supports: Duo Bregman and Duo Jensen Divergences.

By calculating the Kullback–Leibler divergence between two probability measures belonging to different exponential families dominated by the same measure, we obtain a formula that generalizes the ordinary Fenchel–Young divergence. Inspired by this formula, we define the duo Fenchel–Young divergence and report a majorization condition on its pair of strictly convex generators, which guarantees that this divergence is always non-negative. The duo Fenchel–Young divergence is also equivalent to a duo Bregman divergence. We show how to use these duo divergences by calculating the Kullback–Leibler divergence between densities of truncated exponential families with nested supports, and report a formula for the Kullback–Leibler divergence between truncated normal distributions. Finally, we prove that the skewed Bhattacharyya distances between truncated exponential families amount to equivalent skewed duo Jensen divergences.

The Kullback–Leibler Divergence Between Lattice Gaussian Distributions

Discrete normal distributions are defined as the distributions with prescribed means and covariance matrices which maximize entropy on the integer lattice support. The set of discrete normal distributions form an exponential family with cumulant function related to the Riemann theta function. In this paper, we present several formula for common statistical divergences between discrete normal distributions including the Kullback-Leibler divergence. In particular, we describe an efficient approximation technique for calculating the Kullback-Leibler divergence between discrete normal distributions via the R\'enyi $\alpha$-divergences or the projective $\gamma$-divergences.

​Effect of Hydrilla verticillata in Feed on Production Performance and Carcass Characteristics of Thai Native Chickens

Background: Producing Thai native chickens through feed commercial is costly. To reduce the cost, native chicken could be fed with local plants or agricultural waste. Hyrrilla verticillate, a dominate macrophyte in Songkhla Lagoon, contain high crude protein and nitrogen free extract. Therefore, this research studied the use of Hydrilla verticillate in Thai native chicken diet on production performance, carcass characteristics and production cost. Methods: The experiment was executed in a completely randomized design (CRD) with 180 one-day-old commercial Thai native chicks. Four dietary treatments supplemented with 0%, 5%, 10% and 15% Hydrilla verticillata were formulated for three different phases of the experimental chickens: starter (1-4 weeks), grower (5-8 weeks) and finisher (9-12 weeks). Two chickens from each replicate were chosen for the carcass characteristics study. Result: During the starter phase, there was no notable effect shown among the chickens fed with the diet containing Hydrilla verticillata in terms of feed intake (FI) and body weight gain (BWG) (P greater than 0.05). The control chickens had best feed conversion ratio (FCR) (P less than 0.05), while the grower and the finisher periods, provided with different levels of Hydrilla verticillata, revealed no apparent differences (P greater than 0.05) concerning FI, BWG and FCR. Considering carcass characteristics, there were significant differences in live weight (LW) (P less than 0.05). But, no significant differences between treatments for hot carcass percentage, pectoralis majors, pectoralis minorstrip, thighs, wings, drumsticks and total edible carcass (P greater than 0.05). There were also statistical divergences regarding gizzard weight (GW) (P less than 0.05).

Fluctuation theorems with retrodiction rather than reverse processes

Irreversibility is usually captured by a comparison between the process that happens and a corresponding “reverse process.” In the last decades, this comparison has been extensively studied through fluctuation relations. Here, we revisit fluctuation relations from the standpoint, suggested decades ago by Watanabe, that the comparison should involve the prediction and the retrodiction on the unique process, rather than two processes. We identify a necessary and sufficient condition for a retrodictive reading of a fluctuation relation. The retrodictive narrative also brings to the fore the possibility of deriving fluctuation relations based on various statistical divergences, and clarifies some of the traditional assumptions as arising from the choice of a reference prior.

Generative machine learning with tensor networks: Benchmarks on near-term quantum computers

Noisy, intermediate-scale quantum (NISQ) computing devices have become an industrial reality in the last few years, and cloud-based interfaces to these devices are enabling exploration of near-term quantum computing on a range of problems. As NISQ devices are too noisy for many of the algorithms with a known quantum advantage, discovering impactful applications for near-term devices is the subject of intense research interest. We explore quantum-assisted machine learning (QAML) on NISQ devices through the perspective of tensor networks (TNs), which offer a robust platform for designing resource-efficient and expressive machine learning models to be dispatched on quantum devices. In particular, we lay out a framework for designing and optimizing TN-based QAML models using classical techniques, and then compiling these models to be run on quantum hardware, with demonstrations for generative matrix product state (MPS) models. We put forth a generalized canonical form for MPS models that aids in compilation to quantum devices, and demonstrate greedy heuristics for compiling with a given topology and gate set that outperforms known generic methods in terms of the number of entangling gates, e.g., CNOTs, in some cases by an order of magnitude. We present an exactly solvable benchmark problem for assessing the performance of MPS QAML models, and also present an application for the canonical MNIST handwritten digit dataset. The impacts of hardware topology and day-to-day experimental noise fluctuations on model performance are explored by analyzing both raw experimental counts and statistical divergences of inferred distributions. We also present parametric studies of depolarization and readout noise impacts on model performance using hardware simulators.

A class of non-parametric statistical manifolds modelled on Sobolev space

We construct a family of non-parametric (infinite-dimensional) manifolds of finite measures on {mathbb {R}}^d, each containing a smoothly embedded submanifold of probability measures. The manifolds are modelled on a variety of weighted Sobolev spaces, including Hilbert–Sobolev spaces and mixed-norm spaces, and support the Fisher–Rao metric as a weak Riemannian metric. Densities are expressed in terms of a deformed exponential function having linear growth. Unusually for the Sobolev context, and as a consequence of its linear growth, this “lifts” to a nonlinear superposition (Nemytskii) operator that acts continuously on a particular class of mixed-norm model spaces, and on the fixed norm space W^{2,1}; i.e. it maps each of these spaces continuously into itself. In contrast with non-parametric exponential manifolds, the density itself belongs to the model space, and the range of the chart is the whole of this space. Some of the results make essential use of a log-Sobolev embedding theorem, which also sharpens existing results concerning the regularity of statistical divergences on the manifolds. Applications to the stochastic partial differential equations of nonlinear filtering (and hence to the Fokker–Planck equation) are outlined.

A projection pricing model for non-Gaussian financial returns

Stephen LeRoy, Jan Werner and David Luenberger have developed a geometric approach to the capital asset pricing model (CAPM) in terms of projections in a Hilbert space onto a mean–variance efficient frontier. Using this projection method, they were able to elegantly deduce a geometric interpretation of CAPM and factor asset pricing models. In this paper we extend their geometric methods to non-Euclidean divergence geometries. This extension has relevant consequences. First, it permits to deal with higher order moments of the probability distributions since general statistical divergences could encode global information about these distributions as is the case of the entropy. Secondly, orthogonal Euclidean projections and the corresponding least squares problem give place to Riemannian projections onto a possibly curved efficient frontier. Finally, our method is flexible enough to deal with huge families of probability distributions. In particular, there is no need to assume normality of the returns of the financial assets.

A two‐step proximal‐point algorithm for the calculus of divergence‐based estimators in finite mixture models

AbstractEstimators derived from the expectation‐maximization (EM) algorithm are not robust since they are based on the maximization of the likelihood function. We propose an iterative proximal‐point algorithm based on the EM algorithm to minimize a divergence criterion between a mixture model and the unknown distribution that generates the data. The algorithm estimates in each iteration the proportions and the parameters of the mixture components in two separate steps. Resulting estimators are generally robust against outliers and misspecification of the model. Convergence properties of our algorithm are studied. The convergence of the introduced algorithm is discussed on a two‐component Weibull mixture entailing a condition on the initialization of the EM algorithm in order for the latter to converge. Simulations on Gaussian and Weibull mixture models using different statistical divergences are provided to confirm the validity of our work and the robustness of the resulting estimators against outliers in comparison to the EM algorithm. An application to a dataset of velocities of galaxies is also presented. The Canadian Journal of Statistics 47: 392–408; 2019 © 2019 Statistical Society of Canada

Principal Curves for Statistical Divergences and an Application to Finance.

This paper proposes a method for the beta pricing model under the consideration of non-Gaussian returns by means of a generalization of the mean-variance model and the use of principal curves to define a divergence model for the optimization of the pricing model. We rely on the q-exponential model so consider the properties of the divergences which are used to describe the statistical model and fully characterize the behavior of the assets. We derive the minimum divergence portfolio, which generalizes the Markowitz’s (mean-divergence) approach and relying on the information geometrical aspects of the distributions the Capital Asset Pricing Model (CAPM) is then derived under the geometrical characterization of the distributions which model the data, all by the consideration of principal curves approach. We discuss the possibility of integration of our model into an adaptive procedure that can be used for the search of optimum points on finance applications.

The Robustness of Echoic Log-Surprise Auditory Saliency Detection

The concept of saliency describes how relevant a stimulus is for humans. This phenomenon has been studied under different perspectives and modalities, such as audio, visual, or both. It has been employed in intelligent systems to interact with their environment in an attempt to emulate or even outperform human behavior in tasks, such as surveillance and alarm systems or even robotics. In this paper, we focus on the aural modality and our goal consists in measuring the robustness of Echoic log-surprise in comparison with a set of auditory saliency techniques when tested on noisy environments for the task of saliency detection. The acoustic saliency methods that we have analyzed include Kalinli’s saliency model, Bayesian log-surprise, and our proposed algorithm, Echoic log-surprise. This last method combines an unsupervised approach based on the Bayesian log-surprise and the biological concept of echoic or auditory sensory memory by means of a statistical fusion scheme, where the use of different distance metrics or statistical divergences, such as Renyi’s or Jensen-Shannon’s among others, are considered. Additionally, for comparison purposes, we have also compared some classical onset detection techniques, such as those based on voice activity detection or energy thresholding. Results show that Echoic log-surprise outperforms the detection capabilities of the rest of the techniques analyzed in this paper under a great variety of noises and signal-to-noise ratios, corroborating its robustness in noisy environments. In particular, our algorithm with the Jensen-Shannon fusion scheme produces the best F-scores. With the aim of better understanding the behavior of Echoic log-surprise, we have also studied the influence of its control parameters, depth and memory, and their influence at different noise levels.

Asimetrías en los sistemas educativos y en los patrones institucionales: hacia la construcción de un indicador global de capital humano

En este trabajo se presenta una revisión teórica y empírica sobre la relación capital humano y crecimiento económico. Se encuentra que existe una relación del capital humano con otros factores, como la calidad de la educación, la estabilidad política, la confiabilidad del sistema judicial, la corrupción y la burocracia. Igualmente, se presenta un análisis acerca de las consideraciones teóricas del capital humano, resaltando las divergencias estadísticas presentadas en los coeficientes estimados, cuando se toman distintas variables para capturar los efectos de este factor sobre el producto. Además, se hace una revisión teórica sobre la importancia del capital humano en la productividad, y se analiza las asimetrías institucionales con las estadísticas vigentes para 37 países. Por último, se propone y se calcula un Indicador Global de Capital Humano (IGCH).

Laparoscopic versus open hepatectomy approach for regional hepatolithiasis: A meta-analysis

BackgroundIn the present study, we aimed to compare laparoscopic hepatectomy (LH) with open hepatectomy (OH) approach in patients with regional hepatolithiasis (RHL) using meta-analytical techniques. MethodsA systematic review of the literature was performed to identify studies comparing LH and OH in the management of RHL. Operative parameters, postoperative outcomes, and postoperative complications were evaluated. Meta-analysis was performed using Review Manage Version 5.0 software. ResultsNine studies matched the selection criteria, which included 719 patients (LH = 333 and OH = 386). LH was associated with shorter hospital stay (p < 0.00001), earlier oral intake (p < 0.00001), and fewer complications (p = 0.01). No significant statistical divergences were found between the LH and OH groups in terms of operative time (p = 0.48), blood loss (p = 0.07), intraoperative transfusion rate (p = 0.69), initial stone clearance rate (p = 0.33), and postoperative stone recurrence rate (p = 0.23). ConclusionLH for regional hepatolithiasis appears to be a promising treatment modality for patients with intrahepatic stones. However, large numbers of patients and prospective, randomized, controlled studies are required to lead to a more comprehensive comparison of the two procedures.

Ultrasound Image Despeckling Using Stochastic Distance-Based BM3D

Ultrasound image despeckling is an important research field, since it can improve the interpretability of one of the main categories of medical imaging. Many techniques have been tried over the years for ultrasound despeckling, and more recently, a great deal of attention has been focused on patch-based methods, such as non-local means and block-matching collaborative filtering (BM3D). A common idea in these recent methods is the measure of distance between patches, originally proposed as the Euclidean distance, for filtering additive white Gaussian noise. In this paper, we derive new stochastic distances for the Fisher-Tippett distribution, based on well-known statistical divergences, and use them as patch distance measures in a modified version of the BM3D algorithm for despeckling log-compressed ultrasound images. State-of-the-art results in filtering simulated, synthetic, and real ultrasound images confirm the potential of the proposed approach.

On Hölder Projective Divergences

We describe a framework to build distances by measuring the tightness of inequalities and introduce the notion of proper statistical divergences and improper pseudo-divergences. We then consider the Hölder ordinary and reverse inequalities and present two novel classes of Hölder divergences and pseudo-divergences that both encapsulate the special case of the Cauchy–Schwarz divergence. We report closed-form formulas for those statistical dissimilarities when considering distributions belonging to the same exponential family provided that the natural parameter space is a cone (e.g., multivariate Gaussians) or affine (e.g., categorical distributions). Those new classes of Hölder distances are invariant to rescaling and thus do not require distributions to be normalized. Finally, we show how to compute statistical Hölder centroids with respect to those divergences and carry out center-based clustering toy experiments on a set of Gaussian distributions which demonstrate empirically that symmetrized Hölder divergences outperform the symmetric Cauchy–Schwarz divergence.

The minimum S-divergence estimator under continuous models: the Basu–Lindsay approach

Robust inference based on the minimization of statistical divergences has proved to be a useful alternative to the classical maximum likelihood based techniques. Recently Ghosh et al. (2013) proposed a general class of divergence measures for robust statistical inference, named the S-Divergence Family. Ghosh (2014) discussed its asymptotic properties for the discrete model of densities. In the present paper, we develop the asymptotic properties of the proposed minimum S-Divergence estimators under continuous models. Here we use the Basu-Lindsay approach (1994) of smoothing the model densities that, unlike previous approaches, avoids much of the complications of the kernel bandwidth selection. Illustrations are presented to support the performance of the resulting estimators both in terms of efficiency and robustness through extensive simulation studies and real data examples.

Asymptotic Properties of Minimum S-Divergence Estimator for Discrete Models

Robust inference based on the minimization of statistical divergences has proved to be a useful alternative to the classical techniques based on maximum likelihood and related methods. Recently Ghosh et al. (2013) proposed a general class of divergence measures, namely the S-Divergence Family and discussed its usefulness in robust parametric estimation through some numerical illustrations. In this present paper, we develop the asymptotic properties of the proposed minimum S-Divergence estimators under discrete models.