Gaussian Mixture Model Learning Research Articles

Joint unsupervised contrastive learning and robust GMM for text clustering

Text clustering aims to organize a vast collection of documents into meaningful and coherent clusters, thereby facilitating the extraction of valuable insights. While current frameworks for text clustering try to minimize the anisotropy of pre-trained language models through contrastive learning of text embeddings, the approach of treating in-batch samples as negatives is suboptimal. The K-means algorithm offers a way to sample both hard negatives and false negatives. However, relying solely on a single measure of semantic similarity between distributions and using coarse-grained weighting for negative pairs may potentially limit performance. Furthermore, considering the very similar distribution in text clusters due to rich semantics, the Mahalanobis distance-based Gaussian Mixture Model (GMM) is prone to falling into local optima due to one Gaussian model, having a smaller weight, may gradually merging into another during the parameter evaluation by the EM algorithm. To tackle these challenges, we propose a model named JourTC: Joint unsupervised contrastive learning and robust GMM for Text Clustering. In the contrastive learning phase, hard negatives, potential false negatives, and their corresponding global similarity-aware weights are determined through posterior probabilities derived from a Robust GMM (RGMM). This RGMM utilizes the entropy of each individual Gaussian model as a metric and adaptively adjusts the posterior probabilities of samples based on the Gaussian models with both maximum and minimum entropy to diminish the influence of low-entropy Gaussian models. Extensive experiments have shown that JourTC can be seamlessly integrated into existing text clustering frameworks, leading to a notable improvement in accuracy. Our code is publicly available.11http://github.com/nickhcx/JourTC.

A Network Traffic Anomaly Detection Method Based on Gaussian Mixture Model

How can we learn the normal behavior of some communication processes and predict whether a single communication is under attack, with massive network traffic data representing the time costs of each stage in a single communication process? This paper introduces a statistical method for detecting network traffic anomalies using the Gaussian mixture model. There are two aspects to our contributions. First, we show how to learn the normal behavior of a communication process under the assumption that its time costs are generated from the Gaussian mixture model. Secondly, we show that with the learned Gaussian mixture model, we can predict whether a data point is under attack by computing the likelihood that the data point is drawn from the learned Gaussian distribution. The experimental results show that our method reached high accuracy in some cases, while in some other cases that are more complicated, the data point may have more factors and cannot be represented simply by only one Gaussian mixture model.

GAT–GMM: Generative Adversarial Training for Gaussian Mixture Models

Generative adversarial networks (GANs) learn the distribution of observed samples through a zero-sum game between two machine players, a generator and a discriminator. While GANs achieve great success in learning the complex distribution of image, sound, and text data, they perform suboptimally in learning multimodal distribution-learning benchmarks such as Gaussian mixture models (GMMs). In this paper, we propose Generative Adversarial Training for Gaussian Mixture Models (GAT-GMM), a minimax GAN framework for learning GMMs. Motivated by optimal transport theory, we design the zero-sum game in GAT-GMM using a random linear generator and a softmax-based quadratic discriminator architecture, which leads to a nonconvex concave minimax optimization problem. We show that a gradient descent ascent (GDA) method converges to an approximate stationary minimax point of the GAT-GMM optimization problem. In the benchmark case of a mixture of two symmetric, well-separated Gaussians, we further show that this stationary point recovers the true parameters of the underlying GMM. We discuss the application of the proposed GAT-GMM framework for learning GMMs in the distributed federated learning setting, where the widely used expectation-maximization (EM) algorithm can incur great computational and communication costs. On the other hand, we show that GAT-GMM provides a scalable learning approach and a distributed GDA algorithm can still solve the GAT-GMM minimax problem without incurring extra computation costs. We numerically support our theoretical results by performing experiments which show that our minimax framework is successful in centralized learning tasks and can outperform standard EM-type algorithms in the federated setting.

Toward a robust detection of viscous and turbulent flow regions using unsupervised machine learning

We propose an invariant feature space for the detection of viscous-dominated and turbulent regions (i.e., boundary layers and wakes). The developed methodology uses the principal invariants of the strain and rotational rate tensors as input to an unsupervised Machine Learning Gaussian mixture model. The selected feature space is independent of the coordinate frame used to generate the processed data, as it relies on the principal invariants of the strain and rotational rate, which are Galilean invariants. This methodology allows us to identify two distinct flow regions: a viscous-dominated, rotational region (a boundary layer and a wake region) and an inviscid, irrotational region (an outer flow region). We have tested the methodology on a laminar and a turbulent (using Large Eddy Simulation) case for flows past a circular cylinder at Re = 40 and Re = 3900 and a laminar flow around an airfoil at Re=1×105. The simulations have been conducted using a high-order nodal Discontinuous Galerkin Spectral Element Method. The results obtained are analyzed to show that Gaussian mixture clustering provides an effective identification method of viscous-dominated and rotational regions in the flow. We also include comparisons with traditional sensors to show that the proposed clustering does not depend on the selection of an arbitrary threshold, as required when using traditional sensors.

Point Cloud Registration Based on Learning Gaussian Mixture Models With Global-Weighted Local Representations

In the field of point cloud registration, the ability to characterize the point cloud is core to improving the registration performance. Previous methods either convert point clouds as probability density models but ignore the rich feature of point clouds or only extract the local feature of point clouds without considering the global information. They did not fully utilize the point cloud information, so the characterization abilities of these methods are limited. To solve the above problems, we propose a point cloud registration based on learning Gaussian Mixture Models (GMM) with global-weighted local representations. On the one hand, the point cloud is converted to GMM for registration. Unlike discrete point cloud data, GMM is a compact and lightweight representation. On the other hand, we generate GMM by extracting unique local features and global information from the point cloud. The global information is used to weigh the local features. Thus, the resulting GMM is a distribution with global-weighted local feature information representation ability, fully exploring the point cloud’s local and global information. At the same time, we design a learning guide module to directly solve the transformation without following the EM-solving paradigm. Benefiting from the combination of GMM and learning deep information, this formulation greatly improves the ability to characterize point clouds. Our method shows superiority in registration accuracy and generalization performance on synthetic and real-world datasets. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">The source code will be made public</i> .

Grocery Apps and Consumer Purchase Behavior: Application of Gaussian Mixture Model and Multi-Layer Perceptron Algorithm

The purpose of this study is to investigate and compare the popularity of common grocery apps in Hungary as well as Iran. The data were gathered from Iranian and Hungarian users who had at least one online purchase experience using a grocery app. A Gaussian mixture model (GMM) and multi-layer perceptron (MLP) are used as supervised and unsupervised machine learning algorithms with Python programming to cluster customers and predict consumer behavior. The results revealed that Wolt in Hungary and Snappfood in Iran are the most popular grocery apps. Users in Iran are divided into three groups of users of app services and the type of full covariance has higher accuracy compared to the other three types (96%). Meanwhile, we found that the five apps used in Hungary have provided 95% accuracy from the users’ point of view based on the diagonal covariance. The MSE value (overfitting and cross-validation) is less than 0.1 in the MLP algorithm, which shows an acceptable amount of error. The results of overfitting indicate the proper fit of the MLP model. The findings of this study could be important for managers of online businesses. In the clustering section, the accuracy and value of consumer demographic information have been emphasized. Additionally, in the classification and prediction section, a kind of “customization” has been performed with an emphasis on market segmentation. This research used GMM and MLP machine learning algorithms as a creative way to cluster and classify consumers.

A hybrid steady-state evolutionary algorithm using random swaps for Gaussian model-based clustering

A new hybrid evolutionary algorithm (EA) for Gaussian mixture model-based clustering is proposed. The EA is a steady-state method that, in each generation, selects two individuals from a population, creates two offspring using either mutation or crossover, and fine-tunes the offspring using the expectation maximization (EM) algorithm. The offspring compete with their parents for survival into the next generation. The approach proposed uses a random swap, which replaces a component mean with a randomly chosen feature vector as a mutation operator. In the crossover operator, a random component is copied from the source mixture into a destination mixture. Copying in crossover favors components of the source mixture located away from the components of the destination mixture. In computational experiments, the approach was compared to a multiple restarts EM algorithm, a random swap EM method, and a state-of-the-art hybrid evolutionary algorithm for Gaussian mixture model learning on one real and 29 synthetic datasets. The results indicate that, given the same computational budget, the proposed method usually learns mixtures with higher log-likelihoods than other benchmarks competing algorithms. The partitions of data obtained by the method correspond best to the original divisions of datasets into classes.

Low-frequency seismic random noise attenuation based on EPLL-TV under double prior constraints

The expected patch log likelihood (EPLL) method utilizes the learned Gaussian mixture model (GMM) as a prior for white Gaussian noise removal, so that every patch from noisy data is likely under the prior and the global morphology is remained along with denoising. For suppressing the low-frequency seismic random noise having similar waveforms to the signals, we propose an Expected Patch Log Likelihood-Total Variation (EPLL-TV) algorithm, which is constrained by the EPLL prior and the total variation (TV) prior. In the denoising process, the TV prior introduces the structural features of the current data and restrains the random noise by enhancing the gradient sparsity of seismic data. Hence, the EPLL-TV algorithm is able to choose the Gaussian component which is closest to the features of signals from pre-learned general prior of EPLL, although the features in the patch are severely disturbed by the low-frequency noise. Then the seismic events with complex structures can be accurately restored by the chosen Gaussian component, which provides more reliable structures for denoising via the TV prior. We use the alternating direction method of multipliers (ADMM) to solve the proposed optimization model with double constraints and reconstruct the data in Fourier domain to reduce computational complexity. The results of the synthetic and field seismic data demonstrate that the proposed EPLL-TV method is superior to the other methods mentioned in our paper in terms of recovering the seismic events with complex structures and suppressing the low-frequency seismic random noise which has similar waveforms to the signals.

TERMS: textual emotion recognition in multidimensional space.

Microblogs generate a vast amount of data in which users express their emotions regarding almost all aspects of everyday life. Capturing affective content from these context-dependent and subjective texts is a challenging task. We propose an intelligent probabilistic model for textual emotion recognition in multidimensional space (TERMS) that captures the subjective emotional boundaries and contextual information embedded in a text for robust emotion recognition. It is implausible with discrete label assignment;therefore, the model employs a soft assignment by mapping varying emotional perceptions in a multidimensional space and generates them as distributions via the Gaussian mixture model (GMM). To strengthen emotion distributions, TERMS integrates a probabilistic emotion classifier that captures the contextual and linguistic information from texts. The integration of these aspects, the context-aware emotion classifier and the learned GMM parameters provide a complete coverage for accurate emotion recognition. The large-scale experimentation shows that compared to baseline and state-of-the-art models, TERMS achieved better performance in terms of distinguishability, prediction, and classification performance. In addition, TERMS provide insights on emotion classes, the annotation patterns, and the models application in different scenarios.

Bayesian learning of Gaussian mixture model for calculating debris flow exceedance probability

ABSTRACT Probabilistic modelling of debris flow data provides useful information for quantitative risk assessment, such as exceedance probabilities (EPs) of debris flow quantities. This task can defy many classical statistical models because debris flow data are frequently collected over years or even decades, and the nonuniformity of the nature and the complex physical mechanism of debris flows lead to multimodal distribution characteristics of observational data. This paper proposes a Bayesian framework for learning Gaussian mixture model (GMM) of debris flow quantities (e.g. total discharge Q total and maximum impact pressure P max) and calculating their EPs for risk-informed decision making. GMM provides great flexibility to fit observation data, but are intrinsically unidentifiable due to the label switching. These computational difficulties are addressed using Random Gibbs Sampling and Bridge Sampling in the proposed framework, allowing incorporating the statistical uncertainty in GMM parameters into EP estimation. Equations are derived for the proposed approach and are illustrated using Q total and P max data at Jiangjia Ravine, China. Results show that the proposed approach identifies a bivariate GMM of Q total and P max reflecting the multimodal characteristics of the observed data and quantifies the statistical uncertainty of GMM parameters. Incorporating the statistical uncertainty into EP estimation provides robust estimates.

Improved convergence guarantees for learning Gaussian mixture models by EM and gradient EM

We consider the problem of estimating the parameters a Gaussian Mixture Model with K components of known weights, all with an identity covariance matrix. We make two contributions. First, at the population level, we present a sharper analysis of the local convergence of EM and gradient EM, compared to previous works. Assuming a separation of Ω( logK), we prove convergence of both methods to the global optima from an initialization region larger than those of previous works. Specifically, the initial guess of each component can be as far as (almost) half its distance to the nearest Gaussian. This is essentially the largest possible contraction region. Our second contribution are improved sample size requirements for accurate estimation by EM and gradient EM. In previous works, the required number of samples had a quadratic dependence on the maximal separation between the K components, and the resulting error estimate increased linearly with this maximal separation. In this manuscript we show that both quantities depend only logarithmically on the maximal separation.

Fi-score: a novel approach to characterise protein topology and aid in drug discovery studies

Target evaluation is at the centre of rational drug design and biologics development. In order to successfully engineer antibodies, T-cell receptors or small molecules it is necessary to identify and characterise potential binding or contact sites on therapeutically relevant target proteins. Currently, there are numerous challenges in achieving a better docking precision as well as characterising relevant sites. We devised a first-of-its-kind in silico protein fingerprinting approach based on the dihedral angle and B-factor distribution to probe binding sites and sites of structural importance. Our derived Fi-score can be used to classify protein regions or individual structural subsets of interest and the described scoring system could be integrated into other discovery pipelines, such as protein classification databases, or applied to investigate new targets. We further demonstrated how our method can be integrated into machine learning Gaussian mixture models to predict different structural elements. Fi-score, in combination with other biophysical analytical methods depending on the research goals, could help to classify and systematically analyse not only targets but also drug candidates that bind to specific sites. The described methodology could greatly improve pre-screening stage, target selection and drug repurposing efforts in finding other matching targets. HIGHLIGHTS Description and derivation of a first-of-its-kind in silico protein fingerprinting method using B-factors and dihedral angles. Derived Fi-score allows to characterise the whole protein or selected regions of interest. Demonstration how machine learning using Gaussian mixture models on Fi-scores captures and allows to predict functional protein topology elements. Fi-score is a novel method to help evaluate therapeutic targets and engineer effective biologics. Communicated by Ramaswamy H. Sarma

AstroVaDEr: astronomical variational deep embedder for unsupervised morphological classification of galaxies and synthetic image generation

ABSTRACT We present AstroVaDEr (Astronomical Variational Deep Embedder), a variational autoencoder designed to perform unsupervised clustering and synthetic image generation using astronomical imaging catalogues. The model is a convolutional neural network that learns to embed images into a low-dimensional latent space, and simultaneously optimizes a Gaussian Mixture Model (GMM) on the embedded vectors to cluster the training data. By utilizing variational inference, we are able to use the learned GMM as a statistical prior on the latent space to facilitate random sampling and generation of synthetic images. We demonstrate AstroVaDEr’s capabilities by training it on grey-scaled gri images from the Sloan Digital Sky Survey, using a sample of galaxies that are classified by Galaxy Zoo 2. An unsupervised clustering model is found that separates galaxies based on learned morphological features such as axial ratio, surface brightness profile, orientation, and the presence of companions. We use the learned mixture model to generate synthetic images of galaxies based on the morphological profiles of the Gaussian components. AstroVaDEr succeeds in producing a morphological classification scheme from unlabelled data, but unexpectedly places high importance on the presence of companion objects – demonstrating the importance of human interpretation. The network is scalable and flexible, allowing for larger data sets to be classified, or different kinds of imaging data. We also demonstrate the generative properties of the model, which allow for realistic synthetic images of galaxies to be sampled from the learned classification scheme. These can be used to create synthetic image catalogues or to perform image processing tasks such as deblending.

Accurate Prediction of Gene Mutations with Flow Cytometry Immune-Phenotyping By Machine Learning Algorithm

Introduction Identification of gene mutation status prior treatment has improved our capability in risk stratifying acute myeloid leukemia (AML) patients greatly, but it is really a labor-exhausting work to identify these mutations. In this this study, we hypothesize immunophenotyping could predict gene mutation and we aim to develop machine learning algorithms that could predict the AML gene mutation status with the immunophenotype by clinical flow cytometry. Method Retrospective clinical data of patients with AML, including demographic (age &amp; gender), molecular genetics, cytogenetics as well as flow cytometry (FC) data at National Taiwan University Hospital were collected. A total of 529 newly diagnosed de novo AML from 2009 to 2019 enrolled this study. The median age at diagnosis was 58 years (Table 1). In total, 428 NPM1, 415 FLT3, 331 CEBPA and 338 RUNX1 gene testing results and a total of 529 initial diagnostic FC data from these patients were used in developing the gene mutation prediction models. Each FC data sample contained 100,000 cells acquired on FACSCantoII machine 6 fluorescent channels with multiple fluorescent markers. The markers measured are listed in Table 2. There were 19 combinations of markers and fluorescent channels which were served as feature inputs of the machine learning framework. Our proposed machine learning framework can be divided as a phenotype representation learning paradigm and a classification model. To derive the phenotype representation, we trained a multivariate Gaussian Mixture Model (GMM) on the 19-dimension FC data to capture the training data distribution and characteristics in a probabilistic unsupervised manner. Then, a Fisher-scoring method was used to vectorize each sample as a high dimensional representation via differential computation in terms of the learned GMM parameters. This Fisher vectorization method transformed samples to a high dimensional feature space as phenotype vectors. We performed analysis of variance (ANOVA)-based feature selection on these representations which were finally fed into the support vector machine (SVM) classifier. To alleviate the negative effects of imbalance classes in gene mutation identification tasks, we applied synthetic minority oversampling technique (SMOTE) algorithm which augmented the minority class by interpolating samples near support vectors. We train independent SVM models to detect the occurrences of the four gene mutation. The algorithm is evaluated by randomly divided 5-fold cross validation which separates 80% data for training and 20% for testing. Results This gene mutation rate of this cohort for NPM1, FLT3, CEBPA and RUNX1 were 22.2% (95/428), 25.1% (104/415), 20.2% (67/331), and 13.6% (46/338), respectively. The average accuracies (ACC) of the prediction model performance for NPM1, FLT3, CEBPA and RUNX1 were 82.6%, 76.3%, 84.2% and 84.1%, respectively, whereas the area under the ROC curve (AUC) were 77.9%, 63.4%, 80.7% and 67.7%, respectively (Table. 3). Conclusions We demonstrated the potential of the correlation of recurrent AML gene mutation status with immunophenotype of AML through our preliminary gene mutation prediction model. Further study with larger cohorts followed by external validation are needed to further evaluate the feasibility of using machine learning based algorithm as one of triage tools to support physicians in aggressive AML clinical decision before receiving molecular genetic reports. Disclosures Ko: Roche: Honoraria.

Assisted teleoperation in changing environments with a mixture of virtual guides

Haptic guidance is a powerful technique to combine the strengths of humans and autonomous systems for teleoperation. The autonomous system can provide haptic cues to enable the operator to perform precise movements; the operator can interfere with the plan of the autonomous system leveraging his/her superior cognitive capabilities. However, providing haptic cues such that the individual strengths are not impaired is challenging because low forces provide little guidance, whereas strong forces can hinder the operator in realizing his/her plan. Based on variational inference, we learn a Gaussian mixture model (GMM) over trajectories to accomplish a given task. The learned GMM is used to construct a potential field which determines the haptic cues. The potential field smoothly changes during teleoperation based on our updated belief over the plans and their respective phases. Furthermore, new plans are learned online when the operator does not follow any of the proposed plans or after changes in the environment. User studies confirm that our framework helps users perform teleoperation tasks more accurately than without haptic cues and, in some cases, faster. Moreover, we demonstrate the use of our framework to help a subject teleoperate a 7 DoF manipulator in a pick-and-place task.

Image patch prior learning based on random neighbourhood resampling for image denoising

Image patch priors become a popular tool for image denoising. The Gaussian mixture model (GMM) is remarkably effective in modelling natural image patches. However, GMM prior learning using the expectation maximisation (EM) algorithm is sensitive to the initialisation, often leading to low convergence rate of parameter estimation. In this study, a novel sampling method called random neighbourhood resampling (RNR) is proposed to improve the accuracy and efficiency of parameter estimation. An enhanced GMM (EGMM) learning algorithm is further developed by incorporating RNR into the EM algorithm to initialise and update the GMM prior. The learned EGMM prior is applied in the expected patch log-likelihood (EPLL) framework for image denoising. The effectiveness and performance of the proposed RNR and EGMM algorithm are demonstrated via extensive experimental results comparing with the state-of-the-art image denoising methods, the experimental results show the higher PSNR result of the denoised images using the proposed method. Meanwhile, the authors verified that the proposed method can efficiently reduce the time of image denoising compared with the basic EPLL method.

Polynomial Methods in Statistical Inference: Theory and Practice

This survey provides an exposition of a suite of techniques based on the theory of polynomials, collectively referred to as polynomial methods, which have recently been applied to address several challenging problems in statistical inference successfully. Topics including polynomial approximation, polynomial interpolation and majorization, moment space and positive polynomials, orthogonal polynomials and Gaussian quadrature are discussed, with their major probabilistic and statistical applications in property estimation on large domains and learning mixture models. These techniques provide useful tools not only for the design of highly practical algorithms with provable optimality, but also for establishing the fundamental limits of the inference problems through the method of moment matching. The effectiveness of the polynomial method is demonstrated in concrete problems such as entropy and support size estimation, distinct elements problem, and learning Gaussian mixture models.

Expected likelihood based query for active learning of Gaussian mixture models based classifiers

Typically, a supervised machine learning requires sufficient labelled data to achieve satisfying performance. In many domains, however, the labelled data are often expensive to obtain and requires laborious human effort. In order to minimize the labelled data, we propose an active learning method by introducing the expected likelihood as a criterion in the query system, especially for the Gaussian mixture models based classifiers. In this scenario, the expected log-likelihood is calculated for all unlabelled examples in a pool data set, then an instant with the lowest value is chosen to query the object of the label. Experiments were performed to evaluate the effectiveness of the method. Empirically, using the two class artificial data set we observed that the proposed method can reduce significantly the number of labelled examples required to achieve a certain performance compared with passive learning.

Dimensionality Reduction in Learning Gaussian Mixture Models of Movement Primitives for Contextualized Action Selection and Adaptation

Robotic manipulation often requires adaptation to changing environments. Such changes can be represented by a certain number of contextual variables that may be observed or sensed in different manners. When learning and representing robot motion—usually with movement primitives, it is desirable to adapt the learned behaviors to the current context. Moreover, different actions or motions can be considered in the same framework, using contextualization to decide which action applies to which situation. Such frameworks, however, may easily become large dimensional; thus, requiring to reduce the dimensionality of the parameters space, as well as the amount of data needed to generate and improve the model over experience. In this letter, we propose an approach to obtain a generative model from a set of actions that share a common feature. Such feature, namely a contextual variable, is plugged into the model to generate motion. We encode the data with a Gaussian Mixture Model in the parameter space of probabilistic movement primitives, after performing dimensionality reduction on such parameter space. We append the contextual variable to the parameter space and obtain the number of Gaussian components, i.e., different actions in a dataset, through persistent homology. Then, using multimodal Gaussian mixture regression we can retrieve the most likely actions given a contextual situation and execute them. After actions are executed, we use reward-weighted responsibility GMM update the model after each execution. Experimentation in three scenarios shows that the method drastically reduces the dimensionality of the parameter space; thus, implementing both action selection and adaptation to a changing situation in an efficient way.

When do birds of a feather flock together? k-Means, proximity, and conic programming

Given a set of data, one central goal is to group them into clusters based on some notion of similarity between the individual objects. One of the most popular and widely-used approaches is k-means despite the computational hardness to find its global minimum. We study and compare the properties of different convex relaxations by relating them to corresponding proximity conditions, an idea originally introduced by Kumar and Kannan. Using conic duality theory, we present an improved proximity condition under which the Peng-Wei relaxation of k-means recovers the underlying clusters exactly. Our proximity condition improves upon Kumar and Kannan, and is comparable to that of Awashti and Sheffet where proximity conditions are established for projective k-means. In addition, we provide a necessary proximity condition for the exactness of the Peng-Wei relaxation. For the special case of equal cluster sizes, we establish a different and completely localized proximity condition under which the Amini-Levina relaxation yields exact clustering, thereby having addressed an open problem by Awasthi and Sheffet in the balanced case. Our framework is not only deterministic and model-free but also comes with a clear geometric meaning which allows for further analysis and generalization. Moreover, it can be conveniently applied to analyzing various data generative models such as the stochastic ball models and Gaussian mixture models. With this method, we improve the current minimum separation bound for the stochastic ball models and achieve the state-of-the-art results of learning Gaussian mixture models.