Infinite Gaussian Mixture Model Research Articles

A Stochastic Recurrent Encoder Decoder Network for Multistep Probabilistic Wind Power Predictions.

In this article, a stochastic recurrent encoder decoder neural network (SREDNN), which considers latent random variables in its recurrent structures, is developed for the first time for the generative multistep probabilistic wind power predictions (MPWPPs). The SREDNN enables the stochastic recurrent model under the encoder-decoder framework to engage exogenous covariates to produce better MPWPP. The SREDNN consists of five components, the prior network, the inference network, the generative network, the encoder recurrent network, and the decoder recurrent network. The SREDNN is equipped with two critical advantages compared with conventional RNN-based methods. First, the integration over the latent random variable builds an infinite Gaussian mixture model (IGMM) as the observation model, which drastically increases the expressiveness of the wind power distribution. Secondly, hidden states of the SREDNN are updated in a stochastic way, which builds an infinite mixture of the IGMM for describing the ultimate wind power distribution and enables the SREDNN to model complex patterns across wind speed and wind power sequences. Computational experiments are conducted on a dataset of a commercial wind farm having 25 wind turbines (WTs) and two publicly assessable WT datasets to verify the advantages and effectiveness of the SREDNN for MPWPP. Experimental results show that the SREDNN achieves a lower negative form of the continuously ranked probability score (CRPS ∗) as well as a superior sharpness and comparable reliability of prediction intervals by comparing against considered benchmarking models. Results also show the clear benefit gained from considering latent random variables in SREDNN.

Frequency-Control-Aware Probabilistic Load Flow: An Analytical Method

Probabilistic load flow (PLF) calculation, as a fundamental tool to analyze transmission system behavior, has been studied for decades. Despite a variety of available methods, existing PLF approaches rarely take system control into account. However, system control, as an automatic buffer between the fluctuations in random variables and the variations in system variables (e.g., nodal voltages), has a significant impact on the final PLF result. To consider control actions' influence, this paper proposes the first analytical PLF method for the transmission grid that takes into account primary and secondary frequency controls. This method is based on a high-precision linear power flow model, whose precision is even further improved in this paper by an original correction approach. This paper also proves that if the joint probability distribution (JPD) of random variables is expressed by a Gaussian mixture model (GMM), then the JPD of system variables is an infinite GMM. By leveraging this proposition, the proposed method generates the joint PLF of the whole system at the quasi-steady and steady states. The high accuracy and satisfactory efficiency of this method are verified on multiple test systems with different percentages of renewables.

Meta-learning representations for clustering with infinite Gaussian mixture models

Appropriate representations are critical for a better clustering performance. Although many neural network-based clustering methods have been proposed, they do not directly train neural networks to improve the clustering performance. We propose a method that can meta-learn knowledge for clustering from various labeled data and uses the knowledge for clustering unseen unlabeled data. The proposed method trains neural networks to obtain representations such that the clustering performance improves when the representations are clustered by variational Bayesian (VB) inference with an infinite Gaussian mixture model. For the objective function, we propose a continuous approximation of the adjusted Rand index (ARI) by which we can evaluate the clustering performance from soft clustering assignments. Since the approximated ARI and the VB inference procedure are differentiable, we can backpropagate the objective function through the VB inference procedure to train the neural networks. With experiments using text and image datasets, we demonstrate that our proposed method has a higher adjusted Rand index than the existing methods.

Bayesian Nonparametric Classification for Incomplete Data With a High Missing Rate: an Application to Semiconductor Manufacturing Data

During the semiconductor manufacturing process, predicting the yield of the semiconductor is an important problem. Early detection of defective product production in the manufacturing process can save huge production cost. The data generated from the semiconductor manufacturing process have characteristics of non-normal distributions, random missing patterns and high missing rate, which complicate the prediction of the yield. We propose the Dirichlet Process -Naive Bayes model (DPNB) that can simultaneously impute missing values and address classification problems. Since the DPNB is based on the infinite Gaussian mixture model, it can estimate complex data distributions and make predictions for missing datasets with some missing patterns due to nice properties of the Gaussian distribution. The DPNB also performs well for high missing rates since it uses all information of observed components. Experiments on various real datasets including semiconductor manufacturing data show that the DPNB has better performance than state-of-the-art methods in terms of predicting missing values and target variables as percentage of missing values increases.

Traffic Speed Estimation Based on Multi-Source GPS Data and Mixture Model

The traffic speed information of an urban road network is generally estimated using the widely available taxi GPS data. However, taxi usages are preponderantly restricted to areas with high population density, which results in limited spatial coverage of collected taxi GPS data. Moreover, the traffic speeds of taxies are not guaranteed to well represent the traffic speeds of other types of vehicles. In this study, we address these issues by introducing an infinite Gaussian mixture model to estimate traffic speed distribution. The variational inference method is employed to deal with the complicated parameter estimation problem. The proposed mixture model simultaneously combines taxi GPS data, bus GPS data, and mobile phone GPS data, which not only generates the mixed traffic-speed distribution of different types of vehicles but also improves the spatial coverage and the quality of traffic speed estimation. Surprisingly, we find that the incorporation of mobile phone GPS data can considerably improve the model’s ability to sense anomalous traffic conditions. Finally, the mixed traffic-speed distribution is validated using the license plate recognition data.

Mixture models for photometric redshifts

Context. Determining photometric redshifts (photo-zs) of extragalactic sources to a high accuracy is paramount to measure distances in wide-field cosmological experiments. With only photometric information at hand, photo-zs are prone to systematic uncertainties in the intervening extinction and the unknown underlying spectral-energy distribution of different astrophysical sources, leading to degeneracies in the modern machine learning algorithm that impacts the level of accuracy for photo-z estimates. Aims. Here, we aim to resolve these model degeneracies and obtain a clear separation between intrinsic physical properties of astrophysical sources and extrinsic systematics. Furthermore, we aim to have meaningful estimates of the full photo-z probability distribution, and their uncertainties. Methods. We performed a probabilistic photo-z determination using mixture density networks (MDN). The training data set is composed of optical (griz photometric bands) point-spread-function and model magnitudes and extinction measurements from the SDSS-DR15 and WISE mid-infrared (3.4 μm and 4.6 μm) model magnitudes. We used infinite Gaussian mixture models to classify the objects in our data set as stars, galaxies, or quasars, and to determine the number of MDN components to achieve optimal performance. Results. The fraction of objects that are correctly split into the main classes of stars, galaxies, and quasars is 94%. Furthermore, our method improves the bias of photometric redshift estimation (i.e., the mean Δz = (zp − zs)/(1 + zs)) by one order of magnitude compared to the SDSS photo-z, and it decreases the fraction of 3σ outliers (i.e., 3 × rms(Δz) < Δz). The relative, root-mean-square systematic uncertainty in our resulting photo-zs is down to 1.7% for benchmark samples of low-redshift galaxies (zs < 0.5). Conclusions. We have demonstrated the feasibility of machine-learning-based methods that produce full probability distributions for photo-z estimates with a performance that is competitive with state-of-the art techniques. Our method can be applied to wide-field surveys where extinction can vary significantly across the sky and with sparse spectroscopic calibration samples. The code is publicly available.

Indoor Multi-Speaker Localization Based on Bayesian Nonparametrics in the Circular Harmonic Domain

Circular microphone arrays have been used for multi-speaker localization in computational auditory scene analysis, for their high flexibility in sound field analysis, including the generation of frequency-invariant eigenbeams for wideband acoustic sources. However, the localization performance of existing circular harmonic approaches, such as circular harmonics beamformer (CHB) depends strongly on the physical characteristics (such as shape) of sensor arrays, and the level of uncertainties presented in acoustic environments (such as background noise, room reverberation, and the number of sources). These uncertainties may limit the performance or practical application of the speaker localization algorithms. To address these issues, in this paper, we present a new indoor multi-speaker localization method in the circular harmonic domain based on the acoustic holography beamforming (AHB) technique and the Bayesian nonparametrics (BNP) method. More specifically, we use the AHB technique, which combines the delay-and-sum beamforming with acoustic-holography-based virtual sensing, to generate direction of arrival (DOA) measurements in the time-frequency (TF) domain, and then design a BNP algorithm based on the infinite Gaussian mixture model (IGMM) to estimate the DOAs of the individual sources without the prior knowledge about the number of sources. These estimates may degrade in the presence of room reverberation and background noise. To address this issue, we develop a robust TF bin selection and permutation method on the basis of mixture weights, using power, power ratio and local variance estimated at each TF bin. Experiments performed on both simulated and real-data show that our method gives significantly better performance, than four recent baseline methods, in a variety of noise and reverberation levels, in terms of the root-mean-square error (RMSE) of the DOA estimation and the source detecting success rate.

A Robust Infinite Gaussian Mixture Model and Its Application in Fault Detection of Nonlinear Multimode Processes

A Robust Infinite Gaussian Mixture Model and Its Application in Fault Detection of Nonlinear Multimode Processes

Detecting Alzheimer’s Dementia Degree

The diagnosis of Alzheimer’s disease (AD) faces two important issues. They are how to extract the features of the rhythms of patients with AD and how to label them and reveal the degree of dementia in patients. This study defines 14 instantaneous power indicators of dementia judgment through Hilbert marginal spectrum (HMS) from rhythm waves. A warped infinite Gaussian mixture model (WiGMM) is proposed to learn the latent variables of these indicators to detect the degree of dementia. The experimental results show that HMS-based indicators are able to reflect the cognitive function of AD patients. This proposed method has the ability to detect brain cognitive status through a warped transform and Dirichlet process parameter prior to inference.

Coupled IGMM-GANs with Applications to Anomaly Detection in Human Mobility Data

Detecting anomalous activity in human mobility data has a number of applications, including road hazard sensing, telematics-based insurance, and fraud detection in taxi services and ride sharing. In this article, we address two challenges that arise in the study of anomalous human trajectories: (1) a lack of ground truth data on what defines an anomaly and (2) the dependence of existing methods on significant pre-processing and feature engineering. Although generative adversarial networks (GANs) seem like a natural fit for addressing these challenges, we find that existing GAN-based anomaly detection algorithms perform poorly due to their inability to handle multimodal patterns. For this purpose, we introduce an infinite Gaussian mixture model coupled with (bidirectional) GANs—IGMM-GAN—that is able to generate synthetic, yet realistic, human mobility data and simultaneously facilitates multimodal anomaly detection. Through the estimation of a generative probability density on the space of human trajectories, we are able to generate realistic synthetic datasets that can be used to benchmark existing anomaly detection methods. The estimated multimodal density also allows for a natural definition of outlier that we use for detecting anomalous trajectories. We illustrate our methodology and its improvement over existing GAN anomaly detection on several human mobility datasets, along with MNIST.

A Differentially Private Big Data Nonparametric Bayesian Clustering Algorithm in Smart Grid

Smart systems, including smart grid (SG) and Internet of Things (IoT), have been playing a critical role in addressing contemporary issues. Taking full advantage of the big data generated by the smart grid can enhance the system stability and reliability, increase asset utilization, and offer better customer experience. To better support the data-driven smart grid, the machine learning technologies such as cluster analysis can be applied to process the massive data generated in smart grid. However, the process of cluster analysis may cause the disclosure of personal private information. In this paper, to achieve privacy-preserving cluster analysis in smart grid, we propose IDPC, a Differentially Private Clustering algorithm based on the Infinite Gaussian mixture model (IGMM). IDPC uses a combination of nonparametric Bayesian method and differential privacy. The nonparametric Bayesian method allows certain parameters to change along with the data and it is usually adopted in a clustering algorithm without a fixed number of clusters. The Laplace mechanism is used in data releasing process to make IDPC differentially private. We present how to make the nonparametric Bayesian clustering algorithm differentially private by adding Laplace noise. By security analysis and performance evaluation, IDPC is proved to be privacy-preserving as well as efficient.

Fuzzy Transfer Learning Using an Infinite Gaussian Mixture Model and Active Learning

Transfer learning is gaining considerable attention due to its ability to leverage previously acquired knowledge to assist in completing a prediction task in a related domain. Fuzzy transfer learning, which is based on fuzzy system (especially fuzzy rule-based models), has been developed because of its capability to deal with the uncertainty in transfer learning. However, two issues with fuzzy transfer learning have not yet been resolved: choosing an appropriate source domain and efficiently selecting labeled data for the target domain. This paper proposes an innovative method based on fuzzy rules that combines an infinite Gaussian mixture model (IGMM) with active learning to enhance the performance and generalizability of the constructed model. An IGMM is used to identify the data structures in the source and target domains providing a promising solution to the domain selection dilemma. Further, we exploit the interactive query strategy in active learning to correct imbalances in the knowledge to improve the generalizability of fuzzy learning models. Through experiments on synthetic datasets, we demonstrate the rationality of employing an IGMM and the effectiveness of applying an active learning technique. Additional experiments on real-world datasets further support the capabilities of the proposed method in practical situations.

A coupled ETAS-I2GMM point process with applications to seismic fault detection

Epidemic-type aftershock sequence (ETAS) point process is a common model for the occurrence of earthquake events. The ETAS model consists of a stationary background Poisson process modeling spontaneous earthquakes and a triggering kernel representing the space–time-magnitude distribution of aftershocks. Popular nonparametric methods for estimation of the background intensity include histograms and kernel density estimators. While these methods are able to capture local spatial heterogeneity in the intensity of spontaneous events, they do not capture well patterns resulting from fault line structure over larger spatial scales. Here we propose a two-layer infinite Gaussian mixture model for clustering of earthquake events into fault-like groups over intermediate spatial scales. We introduce a Monte Carlo expectation-maximization (EM) algorithm for joint inference of the ETAS-I2GMM model and then apply the model to the Southern California Earthquake Catalog. We illustrate the advantages of the ETAS-I2GMM model in terms of both goodness of fit of the intensity and recovery of fault line clusters in the Community Fault Model 3.0 from earthquake occurrence data.

Channel Clustering and QoS Level Identification Scheme for Multi-Channel Cognitive Radio Networks

The increasing popularity of wireless services and devices necessitates high bandwidth requirements; however, spectrum resources are not only limited but also heavily underutilized. Multiple license channels that support the same levels of QoS are desirable to resolve the problems posed by the scarcity and inefficient use of spectrum resources in multi-channel cognitive radio networks (MCRNs). One reason is that multimedia services and applications have distinct, stringent QoS requirements. However, due to a lack of coordination between primary and secondary users, identifying the QoS levels supported over available licensed channels has proven to be problematic and has yet to be attempted. This article presents a novel Bayesian non-parametric channel clustering scheme, which identifies the QoS levels supported over available license channels. The proposed scheme employs the infinite Gaussian mixture model and collapsed Gibbs sampler to identify the QoS levels from the feature space of the bit rate, packet delivery ratio, and packet delay variation of licensed channels. Moreover, the real measurements of wireless data traces and comparisons with baseline clustering schemes are used to evaluate the performance of the proposed scheme.

Combining ConvNets with hand-crafted features for action recognition based on an HMM-SVM classifier

This paper proposes a new framework for RGB-D-based action recognition that takes advantages of hand-designed features from skeleton data and deeply learned features from depth maps, and exploits effectively both the local and global temporal information. Specifically, depth and skeleton data are firstly augmented for deep learning and making the recognition insensitive to view variance. Secondly, depth sequences are segmented using the handcrafted features based on skeleton joints motion histogram to exploit the local temporal information. All training segments are clustered using an Infinite Gaussian Mixture Model (IGMM) through Bayesian estimation and labelled for training Convolutional Neural Networks (ConvNets) on the depth maps. Thus, a depth sequence can be reliably encoded into a sequence of segment labels. Finally, the sequence of labels is fed into a joint Hidden Markov Model and Support Vector Machine (HMM-SVM) classifier to explore the global temporal information for final recognition. The proposed framework was evaluated on the widely used MSRAction-Pairs, MSRDailyActivity3D and UTD-MHAD datasets and achieved promising results.

Real-time data-driven gait phase detection using ground contact force measurements: Algorithms, platform design and performance

Abstract The world is experiencing an unprecedented, enduring, and pervasive aging process. With more people who need walking assistance, the demand for gait rehabilitation has increased rapidly over the years. Effective gait rehabilitation requires a comprehensive gait analysis, in which gait phase detection plays an important role. Although many specialized sensing systems have been developed for gait monitoring, most existing gait phase detection algorithms rely on significant input from medical professionals, which are subjective, manual and inaccurate. To address these problems, this paper presents a data-driven approach for real-time gait phase detection. The approach combines an infinite Gaussian mixture model (IGMM) to classify different gait phases based on the ground contact force (GCF) measurement, and a parallel particle filter to estimate and update the model parameters. Effective particle sharing mechanisms are further designed to distribute particles among different working nodes judiciously and thus strike a good balance between computational overhead and estimation accuracy. The proposed algorithm is implemented in our gait monitoring and analysis platform developed on a VMware cluster. Its estimation accuracy and computational efficiency are examined through extensive experiments using the data traces collected from both healthy human subjects and patients with neurological disorders.

IGMM-Based Co-Localization of Mobile Users With Ambient Radio Signals

Co-localization of mobile users combines methods of detecting nearby users and providing them interesting and useful services or information. By exploiting the massive use of smartphones, nearby users can be co-localized using only their captured ambient radio signals. In this paper, we propose a real-time co-localization system, in a centralized manner, that leverages co-located users with high accuracy. We exploit the similarity of radio frequency measurements from users’ mobile terminal. We do not require any further information about them. Our co-localization system is based on a nonparametric Bayesian method called infinite Gaussian mixture model that allows the model parameters to change with observed input data. In addition, we propose a modified version of Gibbs sampling technique with an average similarity threshold to better fit user’s group. We design our system in a completely centralized manner. Hence, it enables the network to control and manage the formation of the users’ groups. We first evaluate the performance of our proposal numerically. Then, we carry out an extensive experiment to demonstrate the feasibility, and the efficiency of our approach with data sets from a real-world setting. Results on experiment favor our algorithm over the state-of-the-art community detection-based clustering method.

Unsupervised Profiling of Microglial Arbor Morphologies and Distribution Using a Nonparametric Bayesian Approach

The morphologies of microglial arbors in brain tissue are important indicators of their state of activation, and their spatial distribution indicates patterns of tissue perturbation. This paper presents a non-parametric Bayesian approach to unsupervised quantitative profiling of microglial activation states, and mapping their 3-D spatial distributions across extended brain tissue regions imaged by mosaiced confocal microscopy. Our approach is to perform large-scale automated reconstruction of all the microglia using a learned label-consistent dictionary model for the microglial processes, and extract 136 arbor features per cell. We then fit an infinite Gaussian mixture model with collapsed Gibbs sampling to the arbor features, to discover arbor-morphological classes, their morphological ‘signatures’, and the associated covariance matrices all in an unsupervised manner. This nonparametric approach can handle clusters with arbitrarily shaped covariance matrices, perform automatic model selection, handle unbalanced clusters, and detect overlapping classes, without the need to specify the number of clusters. The results can be used for diverse investigational purposes, including: population-scale modeling, automated discovery of arbor classes and their distinguishing quantitative signatures and features, quantifying the nature and extent of the cellular heterogeneity, and spatial mapping of cellular arbor morphologies, and mapping tissue alterations.

An infinite Gaussian mixture model with its application in hyperspectral unmixing

Spectral unmixing is a critical issue in multi-spectral data processing, which has the ability to identify the constituent components of a pixel. Most of the hyperspectral unmixing current methods are based on Linear Mixture Model (LMM) and have been widely used in many scenarios. However, both the noise contained in the LMM and the requirement of essential prior knowledge strongly limit their practical applications. In order to address these issues, this paper proposes an iterative approach named CBIGMM. It utilizes infinite Gaussian mixture model to describe the hyperspectral data, which is robust to the noise due to the intrinsic randomness of the Gaussian components; and extracts the endmembers and their corresponding abundance in a fully unsupervised way without prior knowledge. A set of experiment is conducted on both synthetic and real data set from pesticide-contaminated vegetables. The results and analyses show CBIGMM outperforms other methods in addressing hyperspectral unmixing problem.

Ensemble Registration of Multisensor Images by a Variational Bayesian Approach

In this paper, a novel image registration method, called ensemble image registration, is proposed. We use an infinite Gaussian mixture model (IGMM), which is based on a joint Gaussian mixture model and a Dirichlet process (DP), to model the joint intensity scatter plot (JISP) of the unregistered images. The DP is the cornerstone of nonparametric Bayesian statistics, and has capability of determining a proper number of mixing components. To simultaneously register a group of images, the cost function of reducing the dispersion in the JISP is preformed by a Bayesian method using IGMM. A variational Bayesian framework is developed to inference the posterior distribution of the parameters in the IGMM. To evaluate the performance of the proposed method, experiments of ensemble image registration are presented. The results show that the proposed method has improved performance compared with conventional methods.