Gaussian Mixture Density Research Articles

Mixtures of multivariate Gaussians

This article discusses the approximation of probability densities by mixtures of Gaussian densities. The Kullback-Leibler divergence is used as a measure between densities, followed by applications of the EM algorithm. The conditions under which we study these questions are motivated by approximations introduced in non-linear Kalman-type filtering.

Multivariate, heteroscedastic empirical Bayes via nonparametric maximum likelihood

Abstract Multivariate, heteroscedastic errors complicate statistical inference in many large-scale denoizing problems. Empirical Bayes is attractive in such settings, but standard parametric approaches rest on assumptions about the form of the prior distribution which can be hard to justify and which introduce unnecessary tuning parameters. We extend the nonparametric maximum-likelihood estimator (NPMLE) for Gaussian location mixture densities to allow for multivariate, heteroscedastic errors. NPMLEs estimate an arbitrary prior by solving an infinite-dimensional, convex optimization problem; we show that this convex optimization problem can be tractably approximated by a finite-dimensional version. The empirical Bayes posterior means based on an NPMLE have low regret, meaning they closely target the oracle posterior means one would compute with the true prior in hand. We prove an oracle inequality implying that the empirical Bayes estimator performs at nearly the optimal level (up to logarithmic factors) for denoizing without prior knowledge. We provide finite-sample bounds on the average Hellinger accuracy of an NPMLE for estimating the marginal densities of the observations. We also demonstrate the adaptive and nearly optimal properties of NPMLEs for deconvolution. We apply our method to two denoizing problems in astronomy and to two hierarchical linear modelling problems in social science and biology.

Uncertainty-aware pedestrian trajectory prediction via distributional diffusion

Tremendous efforts have been put forth on predicting pedestrian trajectory with generative models to accommodate uncertainty and multi-modality in human behaviors. An individual’s inherent uncertainty, e.g., change of destination, can be masked by complex patterns resulting from the movements of interacting pedestrians. However, latent variable-based generative models often entangle such uncertainty with complexity, leading to limited either latent expressivity or predictive diversity. In this work, we propose to separately model these two factors by implicitly deriving a flexible latent representation to capture intricate pedestrian movements, while integrating predictive uncertainty of individuals with explicit bivariate Gaussian mixture densities over their future locations. More specifically, we present a model-agnostic uncertainty-aware pedestrian trajectory prediction framework, parameterizing sufficient statistics for the mixture of Gaussians that jointly comprise the multi-modal trajectories. We further estimate these parameters of interest by approximating a denoising process that progressively recovers pedestrian movements from noise. Unlike previous studies, we translate the predictive stochasticity to explicit distributions, allowing it to readily generate plausible future trajectories indicating individuals’ self-uncertainty. Moreover, our framework is compatible with different neural net architectures. We empirically show the performance gains over state-of-the-art even with lighter backbones, across most scenes on two public benchmarks.

Meta model-based and cross entropy-based importance sampling algorithms for efficiently solving system failure probability function

The multi-mode system failure probability function (SFPF) can quantify how the distribution parameters of the random input vector affect the system safety and decouple the system reliability-based design optimization model. However, for a problem with a time-consuming implicit performance function and rare failure domain, efficiently solving the SFPF remains significantly challenging. Therefore, in this study, two efficient algorithms are proposed, namely, the meta model-based importance sampling and cross entropy-based importance sampling. The contributions of this study are twofold. The first is constructing a single-loop optimal importance sampling density (SL-OISD) method to decouple the double-loop framework for analyzing the SFPF. The second is establishing two methods to efficiently approximate the SL-OISD and complete the SFPF estimation. The first method is based on the meta model of the system performance function, which is abbreviated as SL-Meta-IS. The second method is based on minimizing the cross entropy between the Gaussian mixture density model and SL-OISD, which is abbreviated as SL-CE-IS. To reduce the number of evaluating the system performance function when approximating the SL-OISD, sampling the SL-OISD, and identifying the state of the samples for completing the SFPF estimation, an adaptive Kriging model of the system performance function is introduced into SL-Meta-IS and SL-CE-IS. Owing to decoupling the double-loop framework into a single-loop framework, replacing the time-consuming system performance function with the economic Kriging model, and employing importance sampling variance reduction techniques to address issues related to the rare failure domain, the proposed SL-Meta-IS and SL-CE-IS methods greatly enhance the efficiency of SFPF estimations. The numerical and practical examples demonstrate that the two proposed methods are superior to the existing algorithms; moreover, the efficiency of SL-CE-IS is higher than that of SL-Meta-IS.

A generalized Budan-Fourier approach to generalized Gaussian and exponential mixtures

<p>In the literature, finite mixture models were described as linear combinations of probability distribution functions having the form $ f(x) = \Lambda \sum\limits_{i = 1}^n w_i f_i(x) $, $ x \in \mathbb{R} $, where $ w_i $ were positive weights, $ \Lambda $ was a suitable normalising constant, and $ f_i(x) $ were given probability density functions. The fact that $ f(x) $ is a probability density function followed naturally in this setting. Our question was: <italic>if we removed the sign condition on the coefficients $ w_i $, how could we ensure that the resulting function was a probability density function?</italic></p><p>The solution that we proposed employed an algorithm which allowed us to determine all zero-crossings of the function $ f(x) $. Consequently, we determined, for any specified set of weights, whether the resulting function possesses <italic>no</italic> such zero-crossings, thus confirming its status as a probability density function.</p><p>In this paper, we constructed such an algorithm which was based on the definition of a suitable sequence of functions and that we called a <italic>generalized Budan-Fourier sequence</italic>; furthermore, we offered theoretical insights into the functioning of the algorithm and illustrated its efficacy through various examples and applications. Special emphasis was placed on generalized Gaussian mixture densities.</p>

Quality-driven Gaussian mixture variational probabilistic network for soft sensor application in PET/PA6 polymerization process

Variational autoencoder has been widely used to build soft sensor models as a deep unsupervised feature extractor in recent years. However, the vanilla VAE employs a single diagonal Gaussian distribution as the approximate posterior in its latent feature space, which may lead to inferior performance when solving complex non-Gaussian multimode modeling problem. To address the issue, this paper proposes a quality-driven Gaussian mixture variational probabilistic network-based (QGMVPN) soft sensor model. First, Gaussian mixture density with diagonal covariance matrices and Householder transformations are implemented to characterize the distributions of non-Gaussian multimode feature. Then, a quality-driven strategy is designed to extract quality-related features and a novel gating mechanism is employed to enhance the nonlinear representation ability. Last, the trained decoder, coupled with the sampling space, is directly applied to soft sensor modeling owing to the special extra input of the decoder. The effectiveness of QGMVPN-based model is demonstrated on two polymerization datasets.

Short-Term Density Forecasting of Low-Voltage Load Using Bernstein-Polynomial Normalizing Flows

The transition to a fully renewable energy grid requires better forecasting of demand at the low-voltage level to increase efficiency and ensure reliable control. However, high fluctuations and increasing electrification cause huge forecast variability, not reflected in traditional point estimates. Probabilistic load forecasts take future uncertainties into account and thus allow more informed decision-making for the planning and operation of low-carbon energy systems. We propose an approach for flexible conditional density forecasting of short-term load based on Bernstein polynomial normalizing flows, where a neural network controls the parameters of the flow. In an empirical study with 363 smart meter customers, our density predictions compare favorably against Gaussian and Gaussian mixture densities. Also, they outperform a non-parametric approach based on the pinball loss for 24h-ahead load forecasting for two different neural network architectures.

Voltage and temperature information ensembled probabilistic battery health evaluation via deep Gaussian mixture density network

Accurate state-of-health (SOH) estimation of lithium-ion batteries is crucial for ensuring the safety and reliability of electric vehicles. In this paper, a data-driven method is proposed for realizing accurate and robust SOH estimation with uncertainty measures. First, a set of battery health features are proposed based on the original and differential capacity–voltage (Q–V) and temperature–voltage (T–V) curves for depicting the multi-timescale battery ageing behavior. Four metrics, including the mean, variance, Shannon entropy, and fuzzy entropy of the original and processed Q–V/T–V sequences, are considered. The proposed features demonstrate high correlations to battery SOH. Next, a deep learning-assisted Gaussian mixture density network is developed to fuse the proposed features and generate the conditional probability distribution of battery SOH. Finally, a comprehensive computational study is conducted based on three datasets, which contain batteries of different chemistries and operated under different conditions. Results verify that the proposed method can generate an accurate and robust SOH estimation as well as provide the probability measure. The proposed method also presents superior performance to a set of benchmarking methods.

Extracting prime protein targets as possible drug candidates: machine learning evaluation

Extracting “high ranking” or “prime protein targets” (PPTs) as potent MRSA drug candidates from a given set of ligands is a key challenge in efficient molecular docking. This study combines protein-versus-ligand matching molecular docking (MD) data extracted from 10 independent molecular docking (MD) evaluations — ADFR, DOCK, Gemdock, Ledock, Plants, Psovina, Quickvina2, smina, vina, and vinaxb to identify top MRSA drug candidates. Twenty-nine active protein targets (APT) from the enhanced DUD-E repository (http://DUD-E.decoys.org) are matched against 1040 ligands using “forward modeling” machine learning for initial “data mining and modeling” (DDM) to extract PPTs and the corresponding high affinity ligands (HALs). K-means clustering (KMC) is then performed on 400 ligands matched against 29 PTs, with each cluster accommodating HALs, and the corresponding PPTs. Performance of KMC is then validated against randomly chosen head, tail, and middle active ligands (ALs). KMC outcomes have been validated against two other clustering methods, namely, Gaussian mixture model (GMM) and density based spatial clustering of applications with noise (DBSCAN). While GMM shows similar results as with KMC, DBSCAN has failed to yield more than one cluster and handle the noise (outliers), thus affirming the choice of KMC or GMM. Databases obtained from ADFR to mine PPTs are then ranked according to the number of the corresponding HAL-PPT combinations (HPC) inside the derived clusters, an approach called “reverse modeling” (RM). From the set of 29 PTs studied, RM predicts high fidelity of 5 PPTs (17%) that bind with 76 out of 400, i.e., 19% ligands leading to a prediction of next-generation MRSA drug candidates: PPT2 (average HPC is 41.1%) is the top choice, followed by PPT14 (average HPC 25.46%), and then PPT15 (average HPC 23.12%). This algorithm can be generically implemented irrespective of pathogenic forms and is particularly effective for sparse data.Graphical

Probabilistic Intraday PV Power Forecast Using Ensembles of Deep Gaussian Mixture Density Networks

There is a growing interest of estimating the inherent uncertainty of photovoltaic (PV) power forecasts with probability forecasting methods to mitigate accompanying risks for system operators. This study aims to advance the field of probabilistic PV power forecast by introducing and extending deep Gaussian mixture density networks (MDNs). Using the sum of the weighted negative log likelihood of multiple Gaussian distributions as a minimizing objective, MDNs can estimate flexible uncertainty distributions with nearly all neural network structures. Thus, the advantages of advances in machine learning, in this case deep neural networks, can be exploited. To account for the epistemic (e.g., model) uncertainty as well, this study applies two ensemble approaches to MDNs. This is particularly relevant for industrial applications, as there is often no extensive (manual) adjustment of the forecast model structure for each site, and only a limited amount of training data are available during commissioning. The results of this study suggest that already seven days of training data are sufficient to generate significant improvements of 23.9% in forecasting quality measured by normalized continuous ranked probability score (NCRPS) compared to the reference case. Furthermore, the use of multiple Gaussian distributions and ensembles increases the forecast quality relatively by up to 20.5% and 19.5%, respectively.

Jacobi stochastic volatility factor for the LIBOR market model

We propose a new method to efficiently price swap rate derivatives under the LIBOR market model with stochastic volatility and displaced diffusion. This method applies series expansion techniques built around Gaussian (Gram–Charlier) or Gaussian mixture densities to polynomial processes. The standard pricing method for the considered model relies on dynamics freezing to recover a Heston-type model for which analytical formulas are available. This approach is time-consuming, and efficient approximations based on Gram–Charlier expansions have been proposed recently. In this article, we first discuss the fact that for a class of stochastic volatility model, including the Heston one, the classical sufficient condition ensuring the convergence of Gram–Charlier series is not satisfied. Then we propose an approximating model based on a Jacobi process for which we can prove the stability of Gram–Charlier-type expansions. For this approximation, we have been able to prove a strong convergence towards the original model; moreover, we give an estimate of the convergence rate. We also prove a new result on the convergence of the Gram–Charlier series when the volatility factor is not bounded from below. We finally illustrate our convergence results with numerical examples.

Damage Pattern Recognition and Crack Propagation Prediction for Crumb Rubber Concrete Based on Acoustic Emission Techniques

In this study, the clustering method of the concrete matrix rupture and rubber fracture damages as well as the prediction of the ultimate load of crumb rubber concrete using the acoustic emission (AE) technique were investigated. The loading environment of the specimens was a four-point bending load. Six clustering methods including k-means, fuzzy c-means (FCM), self-organizing mapping (SOM), Gaussian mixture model (GMM), hierarchical model, and density peak clustering method were analyzed; the results illustrated that the density peak clustering has the best performance. Next, the optimal clustering algorithm was used to cluster AE signals so as to study the evolution behavior of different damage modes, and the ultimate load of crumb rubber concrete was predicted by an artificial neural network. The results indicated that the combination of AE techniques and appropriate clustering methods such as the density peak clustering method and the artificial neural network could be used as a practical tool for structural health monitoring of crumb rubber concrete.

CGHMM based fault identification for single point diamond flycutting machines

The ultra-precision single point diamond flycutting method is commonly used to produce fine optical surfaces. But the motion errors of machine tool are very weak, it is hard to achieve the error detection for single point diamond flycutting machines. To solve this problem, different measuring data such as vibrations and temperatures of different components were collected synchronously, and then we established the fault identification method based on hidden Markov model (HMM) to analyse the acquired measuring signals. The method includes signal analysis of raw data, fault recognition based on continuous Gaussian mixture density hidden Markov model (CGHMM) and information fusion. It combines the advantages of HMM and information fusion, to identify fault information accurately. Results show that the identification rates of the temperature signals to classification status by GRMS can reach 92% and vibration signals are little related to the surface quality of the workpieces classified by GRMS.

Intelligent random noise modeling by the improved variational autoencoding method and its application to data augmentation

Deep-learning methods facilitate the development of seismic data processing methods; however, they also offer some challenges. The primary challenges are the lack of labeled samples for training, due to heterogeneity in seismic data, expensive acquisition apparatus, and data confidentiality. These problems limit the acquisition of high-quality training data. To solve this problem, we have developed variational autoencoding (VAE) to generate synthetic noise for data augmentation; however, the simplified Kullback-Leibler (KL) distance definition and parameter learning result in the outputs of the original VAE being blurry. To optimize VAE for simulating random desert noise and improve its simulation capability, here we have developed an improved VAE based on KL redefinition and learning parameter replacement. Specifically, we (1) build a training set containing desert random noise samples, (2) redefine the KL distance calculated between two Gaussian mixture densities (rather than two simple Gaussians) because the KL distance plays an important role in the learning accuracy of VAE, and (3) use [Formula: see text] rather than [Formula: see text] to improve the learning efficiency. Statistical analysis indicates that the simulated random noise is statistically indistinguishable from real noise, indicating that our improved VAE is suitable for noise modeling. We also trained a denoising convolutional neural network (DnCNN) using the simulated noise. Data augmentation conducted using the simulated noise improved the effect of DnCNN, proving that our method contributes to data augmentation.

Optimization learning of hidden Markov model using the bacterial foraging optimization algorithm for speech recognition

Nowadays, the speech recognition applications can be found in several activities, and their existence as a field of study and research lasts for a long time. Although, many studies deal with different problems, in security-related areas, biometric identification, access to the Smartphone… Etc. In automatic speech recognition (ASR) systems, hidden Markov models (HMMs) have widely used for modeling the temporal speech signal. In order to optimize HMM parameters (i.e., observation and transition probabilities), iterative algorithms commonly used such as Forward-Backward or Baum-Welch. In this article, we propose to use the bacterial foraging optimization algorithm (BFOA) to enhance HMM with Gaussian mixture densities. As a global optimization algorithm of current interest, BFOA has proven itself for distributed optimization and control. Our experimental results show that the proposed approach yields a significant improvement of the transcription accuracy at signal/noise ratios greater than 15 dB.

Enabling Efficient Model-Free Control of Large-Scale Canals by Exploiting Domain Knowledge

Canals are constructed worldwidely to divert water from rich to arid areas to mitigate water shortages. Since water resource is fairly limited, it is essential to perform canal control efficiently to improve water-delivery performance. A promising solution is to leverage model predictive control (MPC), which calculates the desired canal action at each time step via reliable predictions of the model. However, the predictive model dependence degrades the practicability and the iterative calculation incurs intensive computations, especially for large-scale canals with high-dimensional state and action spaces (curse of dimensionality). This article presents a new canal control model named efficient model-free canal control (EMCC) that obtains control policies in a model-free way via deep reinforcement learning (DRL) and alleviates the curse of dimensionality via domain knowledge (control experience). EMCC adopts the hidden Markov model with Gaussian mixture densities (GMM-HMM) to model canal system dynamics with control experience, and initializes it according to the actual operation data. Besides, we design a reward generator collaborated with GMM-HMM to supervise the reinforcement learning around control experiences to obtain more efficient control policies. We evaluate EMCC via numerical simulations on Chinese largest water-delivery project (SNWTP). Experimental results show that EMCC leads to significant convergence performances compared with crude applications of DRL on large-scale canals, and achieves desired objectives more satisfactorily than MPC and control-experience under two typical water-delivery tasks.

On the nonparametric maximum likelihood estimator for Gaussian location mixture densities with application to Gaussian denoising

We study the nonparametric maximum likelihood estimator (NPMLE) for estimating Gaussian location mixture densities in $d$-dimensions from independent observations. Unlike usual likelihood-based methods for fitting mixtures, NPMLEs are based on convex optimization. We prove finite sample results on the Hellinger accuracy of every NPMLE. Our results imply, in particular, that every NPMLE achieves near parametric risk (up to logarithmic multiplicative factors) when the true density is a discrete Gaussian mixture without any prior information on the number of mixture components. NPMLEs can naturally be used to yield empirical Bayes estimates of the oracle Bayes estimator in the Gaussian denoising problem. We prove bounds for the accuracy of the empirical Bayes estimate as an approximation to the oracle Bayes estimator. Here our results imply that the empirical Bayes estimator performs at nearly the optimal level (up to logarithmic factors) for denoising in clustering situations without any prior knowledge of the number of clusters.

Lower Limb Locomotion Activity Recognition of Healthy Individuals Using Semi-Markov Model and Single Wearable Inertial Sensor.

Lower limb locomotion activity is of great interest in the field of human activity recognition. In this work, a triplet semi-Markov model-based method is proposed to recognize the locomotion activities of healthy individuals when lower limbs move periodically. In the proposed algorithm, the gait phases (or leg phases) are introduced into the hidden states, and Gaussian mixture density is introduced to represent the complex conditioned observation density. The introduced sojourn state forms the semi-Markov structure, which naturally replicates the real transition of activity and gait during motion. Then, batch mode and on-line Expectation-Maximization (EM) algorithms are proposed, respectively, for model training and adaptive on-line recognition. The algorithm is tested on two datasets collected from wearable inertial sensors. The batch mode recognition accuracy reaches up to 95.16%, whereas the adaptive on-line recognition gradually obtains high accuracy after the time required for model updating. Experimental results show an improvement in performance compared to the other competitive algorithms.

A novel multi-odour identification by electronic nose using non-parametric modelling-based feature extraction and time-series classification

The electronic nose (e-nose) is an olfaction system that consists of an array of chemical sensors and effective machine learning algorithms for the detection of various target odours. Feature extraction and classification methods are of great importance in improving the performance of the e-nose system. In this paper, a novel odour identification method is presented. Firstly, we use the kernel-based system modelling approach to extract odour features. Its solution is a series of finite impulse responses which containing discriminant information of different odours. In addition, a parameter optimisation method based on normalised mean square error and information entropy is proposed to optimise the kernel function. The entropy is effective in preventing the finite impulse responses from overfitting. Multi-odour classification is achieved based on Gaussian mixture density hidden Markov model (GMM-HMM) considering the characteristic of the extracted features. Also, parameter selection for GMM-HMM is realised according to BIC index and cross-validation. Then, we validate the performance of the proposed feature extraction method in resistance to noise and compare it with other existed features. The modelling-based feature reached the highest performance even without applying any filtering or smoothing techniques. Finally, we compare the proposed combination of feature extraction and classification algorithms with other approaches. The proposed method outperformed other approaches reaching 93.56% in sensitivity and 98.71% in specificity. The results demonstrate that the proposed method is applicable in e-nose-based odour identification.

Bayesian Nonparametric Joint Mixture Model for Clustering Spatially Correlated Time Series

We develop a Bayesian nonparametric joint mixture model for clustering spatially correlated time series based on both spatial and temporal similarities. In the temporal perspective, the pattern of a time series is flexibly modeled as a mixture of Gaussian processes, with a Dirichlet process (DP) prior over mixture components. In the spatial perspective, the spatial location is incorporated as a feature for clustering, like a time series being incorporated as a feature. Namely, we model the spatial distribution of each cluster as a DP Gaussian mixture density. For the proposed model, the number of clusters does not need to be specified in advance, but rather is automatically determined during the clustering procedure. Moreover, the spatial distribution of each cluster can be flexibly modeled with multiple modes, without determining the number of modes or specifying spatial neighborhood structures in advance. Variational inference is employed for the efficient posterior computation of the proposed model. We validate the proposed model using simulated and real-data examples. Supplementary materials for the article are available online.