Convex Divergence Research Articles

Martingale Methods for Sequential Estimation of Convex Functionals and Divergences

We present a unified technique for sequential estimation of convex divergences between distributions, including integral probability metrics like the kernel maximum mean discrepancy, $\varphi$-divergences like the Kullback-Leibler divergence, and optimal transport costs, such as powers of Wasserstein distances. This is achieved by observing that empirical convex divergences are (partially ordered) reverse submartingales with respect to the exchangeable filtration, coupled with maximal inequalities for such processes. These techniques appear to be complementary and powerful additions to the existing literature on both confidence sequences and convex divergences. We construct an offline-to-sequential device that converts a wide array of existing offline concentration inequalities into time-uniform confidence sequences that can be continuously monitored, providing valid tests or confidence intervals at arbitrary stopping times. The resulting sequential bounds pay only an iterated logarithmic price over the corresponding fixed-time bounds, retaining the same dependence on problem parameters (like dimension or alphabet size if applicable). These results are also applicable to more general convex functionals -- like the negative differential entropy, suprema of empirical processes, and V-Statistics -- and to more general processes satisfying a key leave-one-out property.

Trajectorial dissipation and gradient flow for the relative entropy in Markov chains

We study the temporal dissipation of variance and relative entropy for ergodic Markov Chains in continuous time, and compute explicitly the corresponding dissipation rates. These are identified, as is well known, in the case of the variance in terms of an appropriate Hilbertian norm; and in the case of the relative entropy, in terms of a Dirichlet form which morphs into a version of the familiar Fisher information under conditions of detailed balance. Here we obtain trajectorial versions of these results, valid along almost every path of the random motion and most transparent in the backwards direction of time. Martingale arguments and time reversal play crucial roles, as in the recent work of Karatzas, Schachermayer and Tschiderer for conservative diffusions. Extension are developed to general convex divergences and to countable state-spaces. The steepest descent and gradient flow properties for the variance, the relative entropy, and appropriate generalizations, are studied along with their respective geometries under conditions of detailed balance, leading to a very direct proof for the HWI inequality of Otto and Villani in the present context.

Strongly Convex Divergences.

We consider a sub-class of the f-divergences satisfying a stronger convexity property, which we refer to as strongly convex, or -convex divergences. We derive new and old relationships, based on convexity arguments, between popular f-divergences.

Design and analysis of a high accuracy bidirectional thermal deformable mirror

To realize an efficient, low-cost, and two-way heat-driven deformable mirror, a deformable mirror-integrated structure using a thermoelectric cooler is proposed. The bidirectional heat-driven stroke, thermal field distribution, and deformation were obtained by finite element analysis. At the same time, the error correction effect of the deformed mirror is verified by experiments: the mirror peak value is corrected from 7.087 to 0.386 μm and the root-mean-square value is corrected from 1.347 to 0.081 μm. For four special aberrations—convex divergence, concave divergence, 45-deg astigmatism, and 90-deg astigmatism—the correction error can be controlled below 5%. More importantly, the cost of the deformed mirror structure designed is under 10,000 yuan; the cost is greatly reduced compared to other deformed mirror structures.

Stronger Convergence Results for the Center-Based Fuzzy Clustering With Convex Divergence Measure.

We present a novel alternative convergence theory of the fuzzy C -means (FCM) clustering algorithm with a super-class of the so-called "distance like functions" which emerged from the earlier attempts of unifying the theories of center-based clustering methods. This super-class does not assume the existence of double derivative of the distance measure with respect to the coordinate of the cluster representative (first coordinate in this formulation). The convergence result does not require the separability of the distance measures. Moreover, it provides us with a stronger convergence property comparable (same to be precise, but in terms of the generalized distance measure) to that of the classical FCM with squared Euclidean distance. The crux of the convergence analysis lies in the development of a fundamentally novel mathematical proof of the continuity of the clustering operator even in absence of the closed form upgrading rule, without necessitating the separability and double differentiability of the distance function and still providing us with a convergence result comparable to that of the classical FCM. The implication of our novel proof technique goes way beyond the realm of FCM and provides a general setup for convergence analysis of the similar iterative algorithms.

Unsupervised Learning based Modified C- ICA for Audio Source Separation in Blind Scenario

Separating audio sources from a convolutive mixture of signals from various independent sources is a very fascinating area in personal and professional context. The task of source separation becomes trickier when there is no idea about mixing environment and can be termed as blind audio source separation (BASS). Mixing scenario becomes more complicated when there is a difference between number of audio sources and number of recording microphones, under determined and over determined mixing. The main challenge in BASS is quality of separation and separation speed and the convergence speed gets compromised when separation techniques focused on quality of separation. This work proposed divergence algorithm designed for faster convergence speed along with good quality of separation. Experiments are performed for critically determined audio recording, where number of audio sources is equal to number of microphones and no noise component is taken into consideration. The result advocates that the modified convex divergence algorithm enhance the convergence speed by 20-22% and good quality of separation than conventional convex divergence ICA, Fast ICA, JADE.

Empirical Phi-discrepancies and quasi-empirical likelihood: exponential bounds

We review some recent extensions of the so-called generalized empirical likeli- hood method, when the Kullback distance is replaced by some general convex divergence. We propose to use, instead of empirical likelihood, some regularized form or quasi-empirical likelihood method, corresponding to a convex combination of Kullback and 2 discrepancies. We show that for some adequate choice of the weight in this combination, the corresponding quasi-empirical likelihood is Bartlett-correctable. We also establish some non-asymptotic exponential bounds for the confidence regions obtained by using this method. These bounds are derived via bounds for self-normalized sums in the multivariate case obtained in a pre- vious work by the authors. We also show that this kind of results may be extended to process valued infinite dimensional parameters. In this case some known results about self- normalized processes may be used to control the behavior of generalized empirical likelihood.

General H-theorem and Entropies that Violate the Second Law

H-theorem states that the entropy production is nonnegative and, therefore, the entropy of a closed system should monotonically change in time. In information processing, the entropy production is positive for random transformation of signals (the information processing lemma). Originally, the H-theorem and the information processing lemma were proved for the classical Boltzmann-Gibbs-Shannon entropy and for the correspondent divergence (the relative entropy). Many new entropies and divergences have been proposed during last decades and for all of them the H-theorem is needed. This note proposes a simple and general criterion to check whether the H-theorem is valid for a convex divergence H and demonstrates that some of the popular divergences obey no H-theorem. We consider systems with n states Ai that obey first order kinetics (master equation). A convex function H is a Lyapunov function for all master equations with given equilibrium if and only if its conditional minima properly describe the equilibria of pair transitions Ai ⇌ Aj . This theorem does not depend on the principle of detailed balance and is valid for general Markov kinetics. Elementary analysis of pair equilibria demonstrate that the popular Bregman divergences like Euclidian distance or Itakura-Saito distance in the space of distribution cannot be the universal Lyapunov functions for the first-order kinetics and can increase in Markov processes. Therefore, they violate the second law and the information processing lemma. In particular, for these measures of information (divergences) random manipulation with data may add information to data. The main results are extended to nonlinear generalized mass action law kinetic equations.

On Bregman Distances and Divergences of Probability Measures

This paper introduces scaled Bregman distances of probability distributions which admit nonuniform contributions of observed events. They are introduced in a general form covering not only the distances of discrete and continuous stochastic observations, but also the distances of random processes and signals. It is shown that the scaled Bregman distances extend not only the classical ones studied in the previous literature, but also the information divergence and the related wider class of convex divergences of probability measures. An information-processing theorem is established too, but only in the sense of invariance w.r.t. statistically sufficient transformations and not in the sense of universal monotonicity. Pathological situations where coding can increase the classical Bregman distance are illustrated by a concrete example. In addition to the classical areas of application of the Bregman distances and convex divergences such as recognition, classification, learning, and evaluation of proximity of various features and signals, the paper mentions a new application in 3-D exploratory data analysis. Explicit expressions for the scaled Bregman distances are obtained in general exponential families, with concrete applications in the binomial, Poisson, and Rayleigh families, and in the families of exponential processes such as the Poisson and diffusion processes including the classical examples of the Wiener process and geometric Brownian motion.

Convex Divergence ICA for Blind Source Separation

Independent component analysis (ICA) is vital for unsupervised learning and blind source separation (BSS). The ICA unsupervised learning procedure attempts to demix the observation vectors and identify the salient features or mixture sources. This work presents a novel contrast function for evaluating the dependence among sources. A convex divergence measure is developed by applying the convex functions to the Jensen's inequality. Adjustable with a convexity parameter, this inequality-based divergence measure has a wide range of the steepest descents to reach its minimum value. A convex divergence ICA (C-ICA) is constructed and a nonparametric C-ICA algorithm is derived with different convexity parameters where the non-Gaussianity of source signals is characterized by the Parzen window-based distribution. Experimental results indicate that the specialized C-ICA significantly reduces the number of learning epochs during estimation of the demixing matrix. The convergence speed is improved by using the scaled natural gradient algorithm. Experiments on the BSS of instantaneous, noisy and convolutive mixtures of speech and music signals further demonstrate the superiority of the proposed C-ICA to JADE, Fast-ICA, and the nonparametric ICA based on mutual information.

The α-EM algorithm: surrogate likelihood maximization using α-logarithmic information measures

A new likelihood maximization algorithm called the /spl alpha/-EM algorithm (/spl alpha/-expectation-maximization algorithm) is presented. This algorithm outperforms the traditional or logarithmic EM algorithm in terms of convergence speed for an appropriate range of the design parameter /spl alpha/. The log-EM algorithm is a special case corresponding to /spl alpha/=-1. The main idea behind the /spl alpha/-EM algorithm is to search for an effective surrogate function or a minorizer for the maximization of the observed data's likelihood ratio. The surrogate function adopted in this paper is based upon the /spl alpha/-logarithm which is related to the convex divergence. The convergence speed of the /spl alpha/-EM algorithm is theoretically analyzed through /spl alpha/-dependent update matrices and illustrated by numerical simulations. Finally, general guidelines for using the /spl alpha/-logarithmic methods are given. The choice of alternative surrogate functions is also discussed.