Process Gaussian Mixture Research Articles

Enabling Virtual Met Masts for wind energy applications through machine learning-methods

As wind is the basis of all wind energy projects, a precise knowledge about its availability is needed. For an analysis of the site-specific wind conditions, Virtual Meteorological Masts (VMMs) are frequently used. VMMs make use of site calibrated numerical data to provide precise wind estimates during all phases of a wind energy project. Typically, numerical data are used for the long-term correlation that is required for estimating the yield of new wind farm projects. However, VMMs can also be used to fill data gaps or during the operational phase as an additional reference data set to detect degrading sensors. The value of a VMM directly depends on its ability and precision to reproduce site-specific environmental conditions. Commonly, linear regression is used as state of the art to correct reference data to the site-specific conditions. In this study, a framework of 10 different machine-learning methods is tested to investigated the benefit of more advanced methods on two offshore and one onshore site. We find significantly improving correlations between the VMMs and the reference data when using more advanced methods and present the most promising ones. The K-Nearest Neighbors and AdaBoost regressors show the best results in our study, but Multi-Output Mixture of Gaussian Processes is also very promising. The use of more advanced regression models lead to decreased uncertainties; hence those methods should find its way into industrial applications. The recommended regression models can serve as a starting point for the development of end-user applications and services.

Mixture of multivariate Gaussian processes for classification of irregularly sampled satellite image time-series

Mixture of multivariate Gaussian processes for classification of irregularly sampled satellite image time-series

Mixture of Gaussian Processes Based on Bayesian Optimization

This paper gives a detailed introduction of implementing mixture of Gaussian process (MGP) model and develops its application for Bayesian optimization (BayesOpt). The paper also develops techniques for MGP in finding its mixture components and introduced an alternative gating network based on the Dirichlet distributions. BayesOpt using the resultant MGP model significantly outperforms the one based on Gaussian process regression in terms of optimization efficiency in the test on tuning the hyperparameters in common machine learning algorithms. This indicates the success of the methods, implying a promising future of wider application for MGP model and the BayesOpt based on it.

Efficient Clustering for Continuous Occupancy Mapping Using a Mixture of Gaussian Processes.

This paper proposes a novel method for occupancy map building using a mixture of Gaussian processes. Gaussian processes have proven to be highly flexible and accurate for a robotic occupancy mapping problem, yet the high computational complexity has been a critical barrier for large-scale applications. We consider clustering the data into small, manageable subsets and applying a mixture of Gaussian processes. One of the problems in clustering is that the number of groups is not known a priori, thus requiring inputs from experts. We propose two efficient clustering methods utilizing (1) a Dirichlet process and (2) geometrical information in the context of occupancy mapping. We will show that the Dirichlet process-based clustering can significantly speed up the training step of the Gaussian process and if geometrical features, such as line features, are available, they can further improve the clustering accuracy. We will provide simulation results, analyze the performance and demonstrate the benefits of the proposed methods.

Attention Mechanism Based Mixture of Gaussian Processes

• We connect the attention mechanism and mixture of Gaussian processes • We design two novel mixture of Gaussian processes models based on the attention mechanism. • The proposed models are applicable to the case that the input variable lies on a manifold or a graph. The mixture of Gaussian processes (MGP) is a powerful model, which is able to characterize data generated by a general stochastic process. However, conventional MGPs assume the input variable obeys certain probabilistic distribution, thus cannot effectively handle the case where the input variable lies on a general manifold or a graph. In this paper, we first clarify the relationship between the MGP prediction strategy and the attention mechanism. Based on the attention mechanism, we further design two novel mixture models of Gaussian processes, which do not rely on probabilistic assumptions on the input domain, thus overcoming the difficulty of extending MGP models to manifold or graph. Experimental results on real-world datasets demonstrate the effectiveness of the proposed methods.

Identifying patterns in amyotrophic lateral sclerosis progression from sparse longitudinal data

The clinical presentation of amyotrophic lateral sclerosis (ALS), a fatal neurodegenerative disease, varies widely across patients, making it challenging to determine if potential therapeutics slow progression. We sought to determine whether there were common patterns of disease progression that could aid in the design and analysis of clinical trials. We developed an approach based on a mixture of Gaussian processes to identify clusters of patients sharing similar disease progression patterns, modeling their average trajectories and the variability in each cluster. We show that ALS progression is frequently nonlinear, with periods of stable disease preceded or followed by rapid decline. We also show that our approach can be extended to Alzheimer’s and Parkinson’s diseases. Our results advance the characterization of disease progression of ALS and provide a flexible modeling approach that can be applied to other progressive diseases.

Model selection of Gaussian mixture process and its application

In this article, new penalized likelihood methods are proposed for model selection of Gaussian mixture process. Our methods integrate functional principal component analysis, kernel density regression, and penalized estimation, which can be carried out by EM algorithms. Component selection and parameter estimation are conducted simultaneously. Monte Carlo simulation shows that our methods require less computation and have accurate estimation even with not well separated data. The methods are further applied to a supermarket customer flow data to reveal some interesting patterns of shopping behavior.

Domain Adaptation Mixture of Gaussian Processes for Online Soft Sensor Modeling of Multimode Processes When Sensor Degradation Occurs

Sensor degradation seriously hinders the practical application of soft sensors. To reduce the negative effect of sensor degradation, in this article, we propose a robust domain adaptation mixture of Gaussian processes (DA-MGP) for online soft sensor modeling of multimode processes. Based on the decomposition of industrial data into a group of Gaussian domains, Gaussian domain discrepancy (GDD) is designed for domain adaptation and process mode recognition. After recognizing the process mode based on GDD, a Gaussian domain adaptation is presented to correct the drifted online input data by domain mapping, which can significantly improve the robustness of the soft sensor against sensor degradation. Furthermore, the domain mapping matrix is utilized as a transferred basis function for a local transferred Gaussian process component, which is used for robust soft sensor modeling. Additionally, an online block processing framework is adopted when the DA-MGP-based soft sensor is applied in online quality prediction. Finally, the TE benchmark process and a real industrial polypropylene process are employed to verify the effectiveness of the proposed method. In the designed five cases of sensor degradation, the DA-MGP-based soft sensor shows its strong robustness against sensor degradation.

Conditional functional clustering for longitudinal data with heterogeneous nonlinear patterns

In studies of cognitive aging, it is crucial to distinguish subtypes of longitudinal cognition change while accounting for the effects of given covariates. The longitudinal cognition trajectories and the covariate effects can both be nonlinear with heterogeneous shapes that do not follow a simple parametric form, where flexible functional methods are preferred. However, most functional clustering methods for longitudinal data do not allow controlling for the possible functional effects of covariates. Although traditional mixture-of-experts methods can include covariates and be extended to the functional setting, using nonlinear basis functions, satisfactory parsimonious functional methods required for robust functional coefficient estimation and clustering are still lacking. In this paper we propose a novel latent class functional mixed-effects model in which we assume the covariates have fixed functional effects, and the random curves follow a mixture of Gaussian processes that facilitates a model-based conditional clustering. A transformed penalized B-spline approach is employed for parsimonious modeling and robust model estimation. We propose a new iterative-REML method to choose the penalty parameters in heterogeneous data. The new method is applied to the latest data from the Religious Orders Study and Rush Memory and Aging Project, and four novel subtypes of cognitive changes are identified.

Bayesian mixture of gaussian processes for data association problem

We address the data association problem and propose a Bayesian approach based on a mixture of Gaussian Processes (GPs) having two key components, the assignment probabilities and the GPs. In the proposed approach, the two key components are simultaneously updated according to observations through an efficient Expectation-Maximization (EM) algorithm that we develop. The proposed approach is thus more adaptive to the observations than the existing approaches for data association. To validate the performance of the proposed approach, we provide experimental results with real data sets as well as two synthetic data sets. We also provide a theoretical analysis to show the effectiveness of the Bayesian update.

Learning low‐dimensional structure in house price indices

AbstractHouse price indices (HPIs) are statistical measures of real estate price dynamics in defined geographic regions over defined periods of time. HPIs are important metrics that help policymakers, mortgage lenders, real estate investors, and bank regulators monitor market conditions and manage risk. HPIs that are local, reliable, and timely are essential in understanding connections between housing markets and the broader economy. In this article, we examine the algorithmic construction of Zillow's Home Value Index (ZHVI), an HPI built on black box machine learning algorithms. To provide deeper statistical insight into ZHVI than afforded by its black box construction, we develop a Bayesian generative meta‐model that approximates the black box construction of ZHVI series in 100 metropolitan areas (metros). Each ZHVI series is modeled with a global trend, a finite mixture of Gaussian processes, and a local component. We find that there are three shared dynamic patterns across the 100 markets in our analysis, and we utilize this shared latent structure to forecast ZHVI in each metro 12 months ahead. Our clustering strategy has two advantages: (i) it allows us to construct composite HPIs where member metros are learned from the data rather than predetermined; and (ii) it allows us to estimate the relative contributions of cluster‐level and metro‐specific components to a metro's ZHVI, providing a novel statistical attribution of real estate market dynamics.

Clustering spatio-temporal series of confirmed COVID-19 deaths in Europe

The impact of the COVID-19 pandemic varied significantly across different countries, with important consequences in the definition of control and response strategies. In this work, to investigate the heterogeneity of this crisis, we analyse the spatial patterns of deaths attributed to COVID-19 in several European countries. To this end, we propose a Bayesian nonparametric approach, based on mixture of Gaussian processes coupled with Dirichlet process, to group the COVID-19 mortality curves. The model provides a flexible framework for the analysis of time series data, allowing the inclusion in the clustering procedure of different features of the series, such as spatial correlations, time varying parameters and measurement errors. We evaluate the proposed methodology on the death counts recorded at NUTS-2 regional level for several European countries in the period from March 2020 to February 2021.

Hierarchical Double Dirichlet Process Mixture of Gaussian Processes

We consider an infinite mixture model of Gaussian processes that share mixture components between non-local clusters in data. Meeds and Osindero (2006) use a single Dirichlet process prior to specify a mixture of Gaussian processes using an infinite number of experts. In this paper, we extend this approach to allow for experts to be shared non-locally across the input domain. This is accomplished with a hierarchical double Dirichlet process prior, which builds upon a standard hierarchical Dirichlet process by incorporating local parameters that are unique to each cluster while sharing mixture components between them. We evaluate the model on simulated and real data, showing that sharing Gaussian process components non-locally can yield effective and useful models for richly clustered non-stationary, non-linear data.

Gaussian process mixture model for prediction based on maximum posterior distribution

In this paper, a numerical experiment was proposed, which is based on the Bayesian procedure and Gaussian Process. Gaussian Process prior distribution was supposed as a before evaluating the hyperparameters of the Gaussian Process model via GyPytorch. Finally, the Gaussian process posterior was achieved and then confidence regions were visualized in two-dimensional planes. The Gaussian Process model could perform under common or specified kernel functions, while it depends on the prior information to a great extent.

Development of security systems using DNN and i & x-vector classifiers

The widespread use of biometric systems entails increased interest from cybercriminals aimed at developing attacks to crack them. Thus, the development of biometric identification systems must be carried out taking into account protection against these attacks. The development of new methods and algorithms for identification based on the presentation of randomly generated key features from the biometric base of user standards will help to minimize the disadvantages of the above methods of biometric identification of users. We present an implementation of a security system based on voice identification as an access control key and a verification algorithm developed using MATLAB function blocks that can authenticate a person's identity by his or her voice. Our research has shown an accuracy of 90 % for this user identification system for individual voice characteristics. It has been experimentally proven that traditional MFCCs using DNN and i and x-vector classifiers can achieve good results. The paper considers and analyzes the most well-known approaches from the literature to the problem of user identification by voice: dynamic programming methods, vector quantization, mixtures of Gaussian processes, hidden Markov model. The developed software package for biometric identification of users by voice and the method of forming the user's voice standards implemented in the complex allows reducing the number of errors in identifying users of information systems by voice by an average of 1.5 times. Our proposed system better defines voice recognition in terms of accuracy, security and complexity. The application of the results obtained will improve the security of the identification process in information systems from various attacks.

Bayesian Pure-Tone Audiometry Through Active Learning Under Informed Priors.

Pure-tone audiometry—the process of estimating a person's hearing threshold from “audible” and “inaudible” responses to tones of varying frequency and intensity—is the basis for diagnosing and quantifying hearing loss. By taking a probabilistic modeling approach, both optimal tone selection (in terms of expected information gain) and hearing threshold estimation can be derived through Bayesian inference methods. The performance of probabilistic model-based audiometry methods is directly linked to the quality of the underlying model. In recent years, Gaussian process (GP) models have been shown to provide good results in this context. We present methods to improve the efficiency of GP-based audiometry procedures by improving the underlying model. Instead of a single GP, we propose to use a GP mixture model that can be conditioned on side-information about the subject. The underlying idea is that one can typically distinguish between different types of hearing thresholds, enabling a mixture model to better capture the statistical properties of hearing thresholds among a population. Instead of modeling all hearing thresholds by a single GP, a mixture model allows specific types of hearing thresholds to be modeled by independent GP models. Moreover, the mixing coefficients can be conditioned on side-information such as age and gender, capturing the correlations between age, gender, and hearing threshold. We show how a GP mixture model can be optimized for a specific target population by learning the parameters from a data set containing annotated audiograms. We also derive an optimal tone selection method based on greedy information gain maximization, as well as hearing threshold estimation through Bayesian inference. The proposed models are fitted to a data set containing roughly 176 thousand annotated audiograms collected in the Nordic countries. We compare the predictive accuracies of optimized mixture models of varying sizes with that of an optimized single-GP model. The usefulness of the optimized models is tested in audiometry simulations. Simulation results indicate that an optimized GP mixture model can significantly outperform an optimized single-GP model in terms of predictive accuracy, and leads to significant increases the efficiency of the resulting Bayesian audiometry procedure.

Modulating scalable Gaussian processes for expressive statistical learning

For a learning task, Gaussian process (GP) is interested in learning the statistical relationship between inputs and outputs, since it offers not only the prediction mean but also the associated variability. The vanilla GP however is hard to learn complicated distribution with the property of, e.g., heteroscedastic noise, multi-modality and non-stationarity, from massive data due to the Gaussian marginal and the cubic complexity. To this end, this article studies new scalable GP paradigms including the non-stationary heteroscedastic GP, the mixture of GPs and the latent GP, which introduce additional latent variables to modulate the outputs or inputs in order to learn richer, non-Gaussian statistical representation. Particularly, we resort to different variational inference strategies to arrive at analytical or tighter evidence lower bounds (ELBOs) of the marginal likelihood for efficient and effective model training. Extensive numerical experiments against state-of-the-art GP and neural network (NN) counterparts on various tasks verify the superiority of these scalable modulated GPs, especially the scalable latent GP, for learning diverse data distributions.

Autoregressive density modeling with the Gaussian process mixture transition distribution

We develop a mixture model for transition density approximation, together with soft model selection, in the presence of noisy and heterogeneous nonlinear dynamics. Our model builds on the Gaussian mixture transition distribution (MTD) model for continuous state spaces, extending component means with nonlinear functions that are modeled using Gaussian process (GP) priors. The resulting model flexibly captures nonlinear and heterogeneous lag dependence when several mixture components are active, identifies low‐order nonlinear dependence while inferring relevant lags when few components are active, and averages over multiple and competing single‐lag models to quantify/propagate uncertainty. Sparsity‐inducing priors on the mixture weights aid in selecting a subset of active lags. The hierarchical model specification follows conventions for both GP regression and MTD models, admitting a convenient Gibbs sampling scheme for posterior inference. We demonstrate properties of the proposed model with two simulated and two real time series, emphasizing approximation of lag‐dependent transition densities and model selection. In most cases, the model decisively recovers important features. The proposed model provides a simple, yet flexible framework that preserves useful and distinguishing characteristics of the MTD model class.

Mixture of robust Gaussian processes and its hard-cut EM algorithm with variational bounding approximation

The Gaussian process is a powerful statistical learning model and has been applied widely in nonlinear regression and classification. However, it fails to model multi-modal data from a non-stationary source since a prior Gaussian process is generally stationary. Based on the idea of the mixture of experts, the mixture of Gaussian processes was established to increase the model flexibility. On the other hand, the Gaussian process is also sensitive to outliers and thus robust Gaussian processes have been suggested to own the heavy-tailed property. In practical applications, the datasets may be multi-modal and contain outliers at the same time. In order to overcome these two difficulties together, we propose a mixture of robust Gaussian processes (MRGP) model and establish a precise hard-cut EM algorithm for learning its parameters. Since the exact solving process is intractable due to the fact that non-Gaussian probability density functions of the noises are adopted into the likelihood of the proposed model on the dataset, we employ a variational bounding method to approximate the marginal likelihood functions so that the hard-cut EM algorithm can be implemented effectively. Moreover, we conduct various experiments on both synthetic and real-world datasets to evaluate and compare our proposed MRGP method with several competitive nonlinear regression methods. The experimental results demonstrate that our MRGP model with the hard-cut EM algorithm is much more effective and robust than the competitive nonlinear regression models.

Sequential mixture of Gaussian processes and saddlepoint approximation for reliability-based design optimization of structures

This paper presents an efficient optimization procedure for solving the reliability-based design optimization (RBDO) problem of structures under aleatory uncertainty in material properties and external loads. To reduce the number of structural analysis calls during the optimization process, mixture models of Gaussian processes (MGPs) are constructed for prediction of structural responses. The MGP is used to expand the application of the Gaussian process model (GPM) to large training sets for well covering the input variable space, significantly reducing the training time, and improving the overall accuracy of the regression models. A large training set of the input variables and associated structural responses is first generated and split into independent subsets of similar training samples using the Gaussian mixture model clustering method. The GPM for each subset is then developed to produce a set of independent GPMs that together define the MGP as their weighted average. The weight vector computed for a specified input variable contains the probability that the input variable belongs to the projection of each subset onto the input variable space. To calculate the failure probabilities and their inverse values required during the process of solving the RBDO problem, a novel saddlepoint approximation is proposed based on the first three cumulants of random variables. The original RBDO problem is replaced by a sequential deterministic optimization (SDO) problem in which the MGPs serve as surrogates for the limit-state functions in probabilistic constraints of the RBDO problem. The SDO problem is strategically solved for exploring a promising region that may contain the optimal solution, improving the accuracy of the MGPs in that region, and producing a reliable solution. Two design examples of a truss and a steel frame demonstrate the efficiency of the proposed optimization procedure.