Abstract
The Jeffreys divergence is a renown arithmetic symmetrization of the oriented Kullback–Leibler divergence broadly used in information sciences. Since the Jeffreys divergence between Gaussian mixture models is not available in closed-form, various techniques with advantages and disadvantages have been proposed in the literature to either estimate, approximate, or lower and upper bound this divergence. In this paper, we propose a simple yet fast heuristic to approximate the Jeffreys divergence between two univariate Gaussian mixtures with arbitrary number of components. Our heuristic relies on converting the mixtures into pairs of dually parameterized probability densities belonging to an exponential-polynomial family. To measure with a closed-form formula the goodness of fit between a Gaussian mixture and an exponential-polynomial density approximating it, we generalize the Hyvärinen divergence to -Hyvärinen divergences. In particular, the 2-Hyvärinen divergence allows us to perform model selection by choosing the order of the exponential-polynomial densities used to approximate the mixtures. We experimentally demonstrate that our heuristic to approximate the Jeffreys divergence between mixtures improves over the computational time of stochastic Monte Carlo estimations by several orders of magnitude while approximating the Jeffreys divergence reasonably well, especially when the mixtures have a very small number of modes.
Highlights
Statistical Mixtures and Statistical DivergencesWe consider the problem of approximating the Jeffreys divergence [1] between two finite univariate continuous mixture models [2] m(x) = ∑ik=1 wi pi (x) and m0 (x) = ∑ik=1 wi0 pi0 (x) with continuous component distributions pi ’s and pi00 s defined on a coinciding support X ⊂ R
We first noticed the simple expression of the Jeffreys divergence between densities pθ and pθ 0 of an exponential family using their dual natural and moment parameterizations [22] pθ = pη and pθ 0 = pη : D J [ p θ, p θ 0 ] = ( θ 0 − θ ) > ( η 0 − η ), where η = ∇ F (θ ) and η 0 = ∇ F (θ 0 ) for the cumulant function F (θ ) of the exponential family
This led us to propose a simple and fast heuristic to approximate the Jeffreys divergence between Gaussian mixture models: First, convert a mixture m to a pair of dually parameterized polynomial exponential densities using extensions of the Maximum
Summary
We consider the problem of approximating the Jeffreys divergence [1] between two finite univariate continuous mixture models [2] m(x) = ∑ik=1 wi pi (x) and m0 (x) = ∑ik=1 wi0 pi0 (x) with continuous component distributions pi ’s and pi00 s defined on a coinciding support X ⊂ R. In practice, when calculating the JD between two GMMs, one can either approximate [14,15], estimate [16], or bound [17,18] the KLD between mixtures. Another approach to bypass the computational intractability of calculating the KLD between mixtures consists of designing new types of divergences that admit closedform expressions for mixtures. KDEs may have a large number of components and may potentially exhibit many spurious modes visualized as small bumps when plotting the densities
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
