Gaussian Mixture Estimation Research Articles

Gaussian Mixture Estimation from Lower-Dimensional Data with Application to PET Imaging

In positron emission tomography (PET), the original points of emission are unknown, and the scanners record pairs of photons emitting from those origins and creating lines of response (LORs) in random directions. This presents a latent variable problem, since at least one dimension of relevant information is lost. This can be solved by a statistical approach to image reconstruction—modeling the image as a Gaussian mixture model (GMM). This allows us to obtain a high-quality continuous model that is not computationally demanding and does not require postprocessing. In this paper, we propose a novel method of GMM estimation in the PET setting, directly from lines of response. This approach utilizes some well-known and convenient properties of the Gaussian distribution and the fact that the random slopes of the lines are independent from the points of origin. The expectation–maximization (EM) algorithm that is most commonly used to estimate GMMs in the traditional setting here is adapted to lower-dimensional data. The proposed estimation method is unbiased, and simulations and experiments show that accurate reconstruction on synthetic data is possible from relatively small samples.

Optimal Bayesian estimation of Gaussian mixtures with growing number of components

We study Bayesian estimation of finite mixture models in a general setup where the number of components is unknown and allowed to grow with the sample size. An assumption on growing number of components is a natural one as the degree of heterogeneity present in the sample can grow and new components can arise as sample size increases, allowing full flexibility in modeling the complexity of data. This however will lead to a high-dimensional model which poses great challenges for estimation. We novelly employ the idea of a sample size dependent prior in a Bayesian model and establish a number of important theoretical results. We first show that under mild conditions on the prior, the posterior distribution concentrates around the true mixing distribution at a near optimal rate with respect to the Wasserstein distance. Under a separation condition on the true mixing distribution, we further show that a better and adaptive convergence rate can be achieved, and the number of components can be consistently estimated. Furthermore, we derive optimal convergence rates for the higher-order mixture models where the number of components diverges arbitrarily fast. In addition, we suggest a simple recipe for using Dirichlet process (DP) mixture prior for estimating the finite mixture models and provide theoretical guarantees. In particular, we provide a novel solution for adopting the number of clusters in a DP mixture model as an estimate of the number of components in a finite mixture model. Simulation study and real data applications are carried out demonstrating the utilities of our method.

Probabilistic seismic petrophysical inversion with statistical double-porosity Biot-Rayleigh model

Petrophysical inversion quantitatively extracts reservoir properties from seismic data/attributes, which has gained increasing attention in assisting hydrocarbon reservoir exploration and evaluation. However, given complex reservoirs, its application is limited either due to the adopted rock-physics model or due to the computational demand. We have developed a semianalytic seismic petrophysical inversion method intended for complex reservoirs. The method incorporates the double-porosity Biot-Rayleigh (BR) model into the petrophysical forward operator, to take into account the heterogeneities within reservoir rocks. The inversion process combines statistical rock-physics modeling and Bayesian Gaussian mixture estimation, by which the analytical expression of the results can be achieved with limited computational cost. To better describe pore structure complexities, the inclusion content that constitutes a double-porosity medium of the BR model is considered an unknown variable, which favors the spatial variation of pore structures under complex lithofacies conditions. In addition, the method uses simulated annealing to estimate the inclusion content at a well location to obtain its prior information and takes advantage of the Monte Carlo simulation in rock-physics modeling to stabilize the inversion results. The method is tested and applied to the field data from a carbonate reservoir in southern China, the results of which show that the accuracy of the petrophysical parameters in terms of root-mean-square error are increased by as much as 12% compared with the method based on the Gassmann model.

Optimal estimation and computational limit of low-rank Gaussian mixtures

Structural matrix-variate observations routinely arise in diverse fields such as multilayer network analysis and brain image clustering. While data of this type have been extensively investigated with fruitful outcomes being delivered, the fundamental questions like its statistical optimality and computational limit are largely under-explored. In this paper, we propose a low-rank Gaussian mixture model (LrMM) assuming each matrix-valued observation has a planted low-rank structure. Minimax lower bounds for estimating the underlying low-rank matrix are established allowing a whole range of sample sizes and signal strength. Under a minimal condition on signal strength, referred to as the information-theoretical limit or statistical limit, we prove the minimax optimality of a maximum likelihood estimator which, in general, is computationally infeasible. If the signal is stronger than a certain threshold, called the computational limit, we design a computationally fast estimator based on spectral aggregation and demonstrate its minimax optimality. Moreover, when the signal strength is smaller than the computational limit, we provide evidences based on the low-degree likelihood ratio framework to claim that no polynomial-time algorithm can consistently recover the underlying low-rank matrix. Our results reveal multiple phase transitions in the minimax error rates and the statistical-to-computational gap. Numerical experiments confirm our theoretical findings. We further showcase the merit of our spectral aggregation method on the worldwide food trading dataset.

Optimal estimation of high-dimensional Gaussian location mixtures

This paper studies the optimal rate of estimation in a finite Gaussian location mixture model in high dimensions without separation conditions. We assume that the number of components k is bounded and that the centers lie in a ball of bounded radius, while allowing the dimension d to be as large as the sample size n. Extending the one-dimensional result of Heinrich and Kahn (Ann. Statist. 46 (2018) 2844–2870), we show that the minimax rate of estimating the mixing distribution in Wasserstein distance is Θ((d/n)1/4+n−1/(4k−2)), achieved by an estimator computable in time O(nd2+n5/4). Furthermore, we show that the mixture density can be estimated at the optimal parametric rate Θ(d/n) in Hellinger distance and provide a computationally efficient algorithm to achieve this rate in the special case of k=2. Both the theoretical and methodological development rely on a careful application of the method of moments. Central to our results is the observation that the information geometry of finite Gaussian mixtures is characterized by the moment tensors of the mixing distribution, whose low-rank structure can be exploited to obtain a sharp local entropy bound.

The basins of attraction of the global minimizers of non-convex inverse problems with low-dimensional models in infinite dimension

Abstract Non-convex methods for linear inverse problems with low-dimensional models have emerged as an alternative to convex techniques. We propose a theoretical framework where both finite dimensional and infinite dimensional linear inverse problems can be studied. We show how the size of the basins of attraction of the minimizers of such problems is linked with the number of available measurements. This framework recovers known results about low-rank matrix estimation and off-the-grid sparse spike estimation, and it provides new results for Gaussian mixture estimation from linear measurements.

Uncovering Hidden Factors in Electricity Consumption Based on Gaussian Mixture Estimation

Load characteristics play an essential role in the planning of power generation and distribution. Various undiscovered factors, which could be socioeconomic, geographic, or climatic, make it possible to describe the electricity demand by a multimodal distribution. This letter proposes a novel method based on multimodal distributions to characterize the hidden factors in electricity consumption. Consequently, a new approach is developed to evaluate the impact of the underlying factors of electricity consumption. Some quantifiable and predictable factors are analyzed in developing multimodal distribution to describe the expected demand. Simulations based on synthetic and real-world data have been conducted to demonstrate the usefulness and robustness of the proposed method.

Variational Sparse Bayesian Learning for Estimation of Gaussian Mixture Distributed Wireless Channels.

In this paper, variational sparse Bayesian learning is utilized to estimate the multipath parameters for wireless channels. Due to its flexibility to fit any probability density function (PDF), the Gaussian mixture model (GMM) is introduced to represent the complicated fading phenomena in various communication scenarios. First, the expectation-maximization (EM) algorithm is applied to the parameter initialization. Then, the variational update scheme is proposed and implemented for the channel parameters’ posterior PDF approximation. Finally, in order to prevent the derived channel model from overfitting, an effective pruning criterion is designed to eliminate the virtual multipath components. The numerical results show that the proposed method outperforms the variational Bayesian scheme with Gaussian prior in terms of root mean squared error (RMSE) and selection accuracy of model order.

Optimal estimation of Gaussian mixtures via denoised method of moments

The method of moments (Philos. Trans. R. Soc. Lond. Ser. A 185 (1894) 71–110) is one of the most widely used methods in statistics for parameter estimation, by means of solving the system of equations that match the population and estimated moments. However, in practice and especially for the important case of mixture models, one frequently needs to contend with the difficulties of non-existence or nonuniqueness of statistically meaningful solutions, as well as the high computational cost of solving large polynomial systems. Moreover, theoretical analyses of the method of moments are mainly confined to asymptotic normality style of results established under strong assumptions. This paper considers estimating a $k$-component Gaussian location mixture with a common (possibly unknown) variance parameter. To overcome the aforementioned theoretic and algorithmic hurdles, a crucial step is to denoise the moment estimates by projecting to the truncated moment space (via semidefinite programming) before solving the method of moments equations. Not only does this regularization ensure existence and uniqueness of solutions, it also yields fast solvers by means of Gauss quadrature. Furthermore, by proving new moment comparison theorems in the Wasserstein distance via polynomial interpolation and majorization techniques, we establish the statistical guarantees and adaptive optimality of the proposed procedure, as well as oracle inequality in misspecified models. These results can also be viewed as provable algorithms for generalized method of moments (Econometrica 50 (1982) 1029–1054) which involves nonconvex optimization and lacks theoretical guarantees.

Semiparametric Clustering: A Robust Alternative to Parametric Clustering.

Clustering aims at naturally grouping the data according to the underlying data distribution. The data distribution is often estimated using a parametric or nonparametric model, e.g., Gaussian mixture or kernel density estimation. Compared with nonparametric models, parametric models are statistically stable, i.e., a small perturbation of data points leads to a small change in the estimated density. However, parametric models are highly sensitive to outliers because the data distribution is far away from the parametric assumptions in the presence of outliers. Given a parametric clustering algorithm, this paper shows how to turn this algorithm into a robust one. The idea is to modify the original parametric density into a semiparametric one. The high-density data that form the core of each cluster are modeled with the original parametric density. The low-density data are often far away from the cluster cores and may have an arbitrary shape, thus are modeled using a nonparametric density. A combination of parametric and nonparametric clustering algorithms is used to group the data modeled as a semiparametric density. From the robust statistical point of view, the proposed method has good robustness properties. We test the proposed algorithm on several synthetic and 70 UCI data sets. The results indicate that the semiparametric method could significantly improve the clustering performance.

Erratum: "Image enhancement via MMSE estimation of Gaussian scale mixture with Maxwell density in AWGN"

Erratum: "Image enhancement via MMSE estimation of Gaussian scale mixture with Maxwell density in AWGN"

Equivalent linearization method using Gaussian mixture (GM-ELM) for nonlinear random vibration analysis

A new equivalent linearization method is developed for nonlinear random vibration analysis. The method employs a Gaussian mixture distribution model to approximate the probabilistic distribution of a nonlinear system response. The parameters of the Gaussian mixture model are estimated by an optimization algorithm which requires a few rounds of dynamic analysis of the nonlinear system. Due to properties of the Gaussian mixture distribution model, the proposed Gaussian mixture based equivalent linearization method (GM-ELM) can decompose the non-Gaussian response of a nonlinear system into multiple Gaussian responses of linear single-degree-of-freedom oscillators. Using a probabilistic combination technique, the linear system of GM-ELM can provide the response probability distribution equal to the Gaussian mixture estimation of the nonlinear response distribution. Using the linear system of GM-ELM in conjunction with linear random vibration theories, response statistics such as the mean up-crossing rate and first-passage probability of the nonlinear system can be conveniently computed. In order to facilitate applications of GM-ELM in earthquake engineering practice, a response spectrum formula is also proposed to compute the mean peak response of the nonlinear system by using the elastic response spectra representing the peak responses of the linear single-degree-of-freedom oscillators. Finally, two numerical examples are presented to illustrate and test GM-ELM. The analysis results obtained from GM-ELM are compared with those obtained from the conventional ELM and Monte-Carlo simulation. The supporting source code and data are available for download at https://github.com/ziqidwang/GitHub-GM-ELM-code.git.

Image enhancement via MMSE estimation of Gaussian scale mixture with Maxwell density in AWGN

In optical techniques, noise signal is a classical problem in medical image processing. Recently, there has been considerable interest in using the wavelet transform with Bayesian estimation as a powerful tool for recovering image from noisy data. In wavelet domain, if Bayesian estimator is used for denoising problem, the solution requires a prior knowledge about the distribution of wavelet coefficients. Indeed, wavelet coefficients might be better modeled by super Gaussian density. The super Gaussian density can be generated by Gaussian scale mixture (GSM). So, we present new minimum mean square error (MMSE) estimator for spherically-contoured GSM with Maxwell distribution in additive white Gaussian noise (AWGN). We compare our proposed method to current state-of-the-art method applied on standard test image and we quantify achieved performance improvement.

PCA-based high-dimensional noisy data clustering via control of decision errors

Cluster analysis is an unsupervised learning technique of partitioning objects into several homogeneous groups. If noises are included in the data set, they should be eliminated in the course of clustering. Although soft clustering methods can handle noise-included data, most of them do not give an appropriate guideline for discriminating noise objects from significant objects. We propose a multiple testing procedure to filter out noises and cluster significant objects while simultaneously maintaining the decision error within the target level. To handle high-dimensional data, we reduce the dimension of attributes by using the principal component analysis and model the objects using the Gaussian mixture. The proposed two-phase procedure is effective in noise separation and in estimation of Gaussian mixture. We applied the proposed procedure to two real and two synthetic data sets. Experimental results show that the proposed method works effectively for high-dimensional data.

Diffusion-Based EM Algorithm for Distributed Estimation of Gaussian Mixtures in Wireless Sensor Networks

Distributed estimation of Gaussian mixtures has many applications in wireless sensor network (WSN), and its energy-efficient solution is still challenging. This paper presents a novel diffusion-based EM algorithm for this problem. A diffusion strategy is introduced for acquiring the global statistics in EM algorithm in which each sensor node only needs to communicate its local statistics to its neighboring nodes at each iteration. This improves the existing consensus-based distributed EM algorithm which may need much more communication overhead for consensus, especially in large scale networks. The robustness and scalability of the proposed approach can be achieved by distributed processing in the networks. In addition, we show that the proposed approach can be considered as a stochastic approximation method to find the maximum likelihood estimation for Gaussian mixtures. Simulation results show the efficiency of this approach.

Interactive Learning of Verbal Descriptors Meanings for Face Drawing System

This paper presents an approach for online learning of descriptors meanings in the Face Drawing System with verbal descriptors. The targeted descriptors are those describing the size of the facial features. The users of the system have the possibility to draw faces using linguistic terms and retouch the features of the drawn face using a combination of an adverb and a linguistic descriptor until the desired face is obtained. The system learns interactively the meanings of the terms from the retouches made by the user on the drawn face. For the current user, the system interactively learns the meanings of the descriptors using the proposed algorithm for online learning of Gaussian fuzzy sets approach based on the Maximum Likelihood Estimation (MLE) of Gaussian mixture. The system then adjusts the drawn face according to the new meaning of the descriptors.

Adaptive Noise Mitigation in Impulsive Environment: Application to Power-Line Communications

In this paper, we address the general issue of asynchronous impulsive noise mitigation and its application over coded power-line communications (PLC). As is well known, PLC channels usually suffer from significant degradation due to impulsive interference generated by electrical appliances. The use of a level limiter is a simple and intuitive technique, widely used to mitigate the noxious effect of impulsive noise in these channels. However, the determination of the clipping threshold remains, most of the time, empirical, and the impulse statistics are usually assumed to be known. In a previous study, we proposed an original threshold determination based on signal detection theory, using the well-known false alarm and good detection tradeoff. Here, we further investigate the proposed optimization method in closed form. We compare two tradeoff criteria by means of a receiver operating curves analysis. We clearly show that the resulting decision fits well with encountered bit-error rate performances. We also evaluate the influence of impulsive noise statistics estimation on the performance of the proposed approach. To do so, we assess the robustness of the classical method of moments on Gaussian mixture estimation. It clearly appears that the estimation method is reliable for reasonable values of the impulse occurrence. As one of the main results of this paper, we make a proposal for impulse statistics estimation under ¿severe¿ conditions (i.e., for very rare impulse events, based on impulse detection over corrupted OFDM symbols). Subsequently, we propose a general scheme of automatic impulse mitigation, according to the disturbance ratio of the environment. Finally, the performance of the approach is evaluated over the Home-Plug AV (HPAV) physical layer.

Multi-directional search from the primitive initial point for Gaussian mixture estimation using variational Bayes method

Gaussian mixture model (GMM) is widely used in many applications because it can approximate various forms of probability distributions. In this paper, we are concerned with GMM estimation problem using the variational Bayes (VB) method. In this approach, one can only find local optima because the free energy function of the problem is multimodal. In order to find better solutions, deterministic annealing was recently adapted to the VB method (DAVB method). In this paper, we offer an alternative approach to the DAVB method for GMM estimation problem. We propose a multi-directional search method from the primitive initial point (PIP), which is defined as the solution of the DAVB method at the highest temperature. Investigation on the curvature information of the original (not annealed) free energy function reveals that the PIP is a saddle point. An efficient multi-directional search strategy from the neighborhoods of the PIP is proposed using the eigen-analysis of the Hessian matrix. Numerical experiments using real data sets demonstrate the effectiveness of our method.

Energy-based video tracking using joint target density processing with an application to unmanned aerial vehicle surveillance

Successfully tracking targets in a video sequence has many important applications, including unmanned aerial vehicle (UAV) surveillance. A robust and efficient video tracking algorithm based on the continuous wavelet transform (CWT) is presented, which has been proven to be effective in capturing motion information over multiple frames and excellent in velocity selectivity. The CWT converts target trajectories in a spatio-temporal domain into target energy volumes in a wavenumber–frequency domain. By integrating over different motion parameters, three target energy densities are obtained, which then serve as cost functions for estimating target trajectories and sizes. Because of excellent velocity selectivity, the energy-based tracker has the capability of detecting and tracking targets with a particular velocity range. To best handle target interferences among multiple nearby or crossing targets, a novel joint processing technique using expectation-maximization-based Gaussian mixture estimation is developed. A global nearest neighbourhood algorithm is employed to perform data association and maintain continuous kinematic trajectories. In addition to computer simulations, the developed energy-based algorithm is applied to a UAV surveillance application where multiple vehicles move closely to each other on a multi-lane road.

An effective method for selecting the number of components in density mixtures

This paper proposes a simple and effective method for selecting the number of components in density mixtures. The proposed method is based on a combination of Stein's unbiased risk estimation and the plug-in principle. Also provided by this paper are numerical results comparing several popular methods for Gaussian mixture estimation. These results demonstrate the superior performance of the proposed method.