Generalized Gaussian Mixture Model Research Articles

Design of intelligent behavior analysis software based on speaker identity classification algorithm in microgrid mode

AbstractDigital technology still has a low level of intelligence in the microgrid mode of teaching behavior analysis, resulting in the traditional manual observation and recording stage still being used for speaker identity classification, and the efficiency of teaching behavior analysis is also low. In response to the above issues, the research is based on the teacher‐student analysis method and proposes a dual clustering algorithm based on the general background model Gaussian mixture model for speaker identity classification, thereby realizing the development and design of intelligent behavior analysis software. The research results indicate that the average recall rate of behavior transition points in the classroom teaching discourse corpus of the intelligent behavior analysis software is 89.03%, which is better than traditional analysis methods. Therefore, the intelligent behavior analysis software constructed by the dual clustering algorithm has high effectiveness and practicality. The research proposes a method model and implements intelligent visualization for classroom teaching behavior analysis, improving the efficiency of analyzing current microgrid teaching behavior.

Novel approach for ECG separation using adaptive constrained IVABMGGMM

In this paper, we introduce the constrained independent vector analysis integrated with the bounded multivariate generalized Gaussian mixture model (cIVABMGGMM) to tackle the limitations of independent component analysis (ICA) when applied to multivariate data, accompanied by its adaptive version, the acIVABMGGMM, aimed at alleviating associated constraints. The acIVABMGGMM employs a full covariance matrix that considers feature correlations, effectively addressing the challenges posed by ICA and independent vector analysis (IVA) models when analyzing multivariate data. The innovative acIVABMGGMM framework merges the adaptability inherent in data-driven methods with the capability to manage noise and other artifacts often encountered in model-based approaches. This technique effectively employs prior knowledge to guide the solution, avoiding the imposition of inaccurate constraints. To overcome these challenges, our set of two constrained methods incorporates prior source information into the IVA model, effectively mitigating its limitations in data with a high number of sources. We assess the efficacy of our proposed models through three distinct ECG separation experiments, which include heartbeat separation, fetal ECG extraction, and arrhythmia detection. Notably, the performance metrics demonstrate the superiority of our models over the baseline approaches in the conducted experiments.

Asymptotic Behavior of Adversarial Training in Binary Linear Classification.

Adversarial training using empirical risk minimization (ERM) is the state-of-the-art method for defense against adversarial attacks, that is, against small additive adversarial perturbations applied to test data leading to misclassification. Despite being successful in practice, understanding the generalization properties of adversarial training in classification remains widely open. In this article, we take the first step in this direction by precisely characterizing the robustness of adversarial training in binary linear classification. Specifically, we consider the high-dimensional regime where the model dimension grows with the size of the training set at a constant ratio. Our results provide exact asymptotics for both standard and adversarial test errors under general lq -norm bounded perturbations ( q ≥ 1 ) in both discriminative binary models and generative Gaussian-mixture models with correlated features. We use our sharp error formulae to explain how the adversarial and standard errors depend upon the over-parameterization ratio, the data model, and the attack budget. Finally, by comparing with the robust Bayes estimator, our sharp asymptotics allow us to study the fundamental limits of adversarial training.

Unsupervised Mixture Models on the Edge for Smart Energy Consumption Segmentation with Feature Saliency.

Smart meter datasets have recently transitioned from monthly intervals to one-second granularity, yielding invaluable insights for diverse metering functions. Clustering analysis, a fundamental data mining technique, is extensively applied to discern unique energy consumption patterns. However, the advent of high-resolution smart meter data brings forth formidable challenges, including non-Gaussian data distributions, unknown cluster counts, and varying feature importance within high-dimensional spaces. This article introduces an innovative learning framework integrating the expectation-maximization algorithm with the minimum message length criterion. This unified approach enables concurrent feature and model selection, finely tuned for the proposed bounded asymmetric generalized Gaussian mixture model with feature saliency. Our experiments aim to replicate an efficient smart meter data analysis scenario by incorporating three distinct feature extraction methods. We rigorously validate the clustering efficacy of our proposed algorithm against several state-of-the-art approaches, employing diverse performance metrics across synthetic and real smart meter datasets. The clusters that we identify effectively highlight variations in residential energy consumption, furnishing utility companies with actionable insights for targeted demand reduction efforts. Moreover, we demonstrate our method's robustness and real-world applicability by harnessing Concordia's High-Performance Computing infrastructure. This facilitates efficient energy pattern characterization, particularly within smart meter environments involving edge cloud computing. Finally, we emphasize that our proposed mixture model outperforms three other models in this paper's comparative study. We achieve superior performance compared to the non-bounded variant of the proposed mixture model by an average percentage improvement of 7.828%.

Data-Weighted Multivariate Generalized Gaussian Mixture Model: Application to Point Cloud Robust Registration.

In this paper, a weighted multivariate generalized Gaussian mixture model combined with stochastic optimization is proposed for point cloud registration. The mixture model parameters of the target scene and the scene to be registered are updated iteratively by the fixed point method under the framework of the EM algorithm, and the number of components is determined based on the minimum message length criterion (MML). The KL divergence between these two mixture models is utilized as the loss function for stochastic optimization to find the optimal parameters of the transformation model. The self-built point clouds are used to evaluate the performance of the proposed algorithm on rigid registration. Experiments demonstrate that the algorithm dramatically reduces the impact of noise and outliers and effectively extracts the key features of the data-intensive regions.

ICA and IVA bounded multivariate generalized Gaussian mixture based hidden Markov models

Machine learning (ML), a branch of artificial intelligence (AI), is an area of computational science that is concerned with the analysis and interpretation of patterns and structures in data to enable learning and decision-making without the participation of a human. Hidden Markov models (HMMs), which have been acknowledged for decades but have recently made a significant revival in machine learning, are one of the most impressively powerful probabilistic models. HMMs are frequently employed in machine learning to model heterogeneous time series data. In this paper, we integrate independent component analysis (ICA) and ICA with a bounded multivariate generalized Gaussian mixture model (ICA-BMGGMM) into the HMM approach. One limitation of ICA is that it assumes the sources to be independent from each other. This assumption can be relaxed by combining independent vectors analysis (IVA) and IVA with the BMGGMM (IVA-BMGGMM) into the HMM approach to improve their modeling capability. We validate our proposed models using a variety of applications, such as human action recognition, speech recognition, and energy disaggregation. The results presented in the paper demonstrate the effectiveness of the proposed approaches for modeling different types of data. These data include KTH and Weizmann datasets for human action recognition, TIMIT and SDR for speech recognition, REDD dataset for energy disaggregation and EEG dataset for elliptic seizure classification. For all conducted experiments, our proposed models outperform other comparing models for all performance metrics such as accuracy, sensitivity, and precision. The best detection results were found using the IVABMGGMM-HMM for the reported experiments.

Sparse and smooth functional data clustering

A new model-based procedure is developed for sparse clustering of functional data that aims to classify a sample of curves into homogeneous groups while jointly detecting the most informative portions of the domain. The proposed method is referred to as sparse and smooth functional clustering (SaS-Funclust) and relies on a general functional Gaussian mixture model whose parameters are estimated by maximizing a log-likelihood function penalized with a functional adaptive pairwise fusion penalty and a roughness penalty. The former allows identifying the noninformative portion of the domain by shrinking the means of separated clusters to some common values, whereas the latter improves the interpretability by imposing some degree of smoothing to the estimated cluster means. The model is estimated via an expectation-conditional maximization algorithm paired with a cross-validation procedure. Through a Monte Carlo simulation study, the SaS-Funclust method is shown to outperform other methods that already appeared in the literature, both in terms of clustering performance and interpretability. Finally, three real-data examples are presented to demonstrate the favourable performance of the proposed method. The SaS-Funclust method is implemented in the R package sasfunclust, available on CRAN.

Bounded multivariate generalized Gaussian mixture model using ICA and IVA

Bounded multivariate generalized Gaussian mixture model using ICA and IVA

Image super-resolution with PCA reduced generalized Gaussian mixture models in materials science

Single Image Super-Resolution algorithms based on patches have been noticed and widely used over the past decade. Recently, generalized Gaussian mixture models (GGMMs) have been shown to be a suitable tool for many image processing problems because of the flexible shape parameter. In this work, we introduce a supervised GGMM-based approach for super-resolution of two- and three-dimensional images, in particular materials images. We first propose to use a joint GGMM learned from concatenated vectors of high- and low-resolution training patches. For each low-resolution patch, we compute the minimum mean square error (MMSE) estimator and generate the high-resolution image by averaging these estimates. We select the MMSE approach using GGMM as the method is invariant to affine contrast change and also invariant to a linear super-resolution operator. Unfortunately, the large dimension of the concatenated high- and low-resolution patches leads to instabilities and an intractable computational effort when estimating the parameters of the GGMM. Thus, we propose to combine a GGMM with a principal component analysis and derive an EM algorithm for estimating the parameters of the arising model. We demonstrate the performance of our model by numerical examples on synthetic and real images of material microstructure.

Graph representation learning based on deep generative gaussian mixture models

Graph representation learning is an effective tool for facilitating graph analysis with machine learning methods. Most GNNs, including Graph Convolutional Networks (GCN), Graph Recurrent Neural Networks (GRNN), and Graph Auto-Encoders (GAE), employ vectors to represent nodes in a deterministic way without exploiting the uncertainty in hidden variables. Deep generative models are combined with GAE in the Variational Graph Auto-Encoder (VGAE) framework to address this issue. While traditional VGAE-based methods can capture hidden and hierarchical dependencies in latent spaces, they are limited by the data’s multimodality. Here, we propose the Gaussian Mixture Model (GMM) to model the prior distribution in VGAE. Furthermore, an adversarial regularization is incorporated into the proposed approach to ensure the fruitful impact of the latent representations on the results. We demonstrate the performance of the proposed method on clustering and link prediction tasks. Our experimental results on real datasets show remarkable performance compared to state-of-the-art methods.

An Estimation of Distribution Algorithm Based on Variational Bayesian for Point-Set Registration

Point-set registration is widely used in computer vision and pattern recognition. However, it has become a challenging problem since the current registration algorithms suffer from the complexities of the point-set distributions. To solve this problem, we propose a robust registration algorithm based on the estimation of distribution algorithm (EDA) to optimize the complex distributions from a global search mechanism. We propose an EDA probability model based on the asymmetric generalized Gaussian mixture model, which describes the area in the solution space as comprehensively as possible and constructs a probability model of complex distribution points, especially for missing and outliers. We propose a transformation and a Gaussian evolution strategy in the selection mechanism of EDA to process the deformation, rotation, and denoising of selected dominant individuals. Considering the complexity of the model, we choose to optimize from the perspective of variational Bayesian, and introduce a prior probability distribution through local variation to reinforce the convergence of the algorithm in dealing with complex point sets. In addition, a local search mechanism based on the simulated annealing algorithm is added to realize the coarse-to-fine registration. Experimental results show that our method has the best robustness compared with the state-of-the-art registration algorithms.

Defect classification on semiconductor wafers using Fisher vector and visual vocabularies coding

The main trend in the classification of defects in semiconductor wafers from scanning electron microscopy images is the use of convolutional neural networks. These methods allow automatic feature extraction and supply high accuracy. However, they have limitations since they need a large number of samples in the training phase and are computationally expensive in general. Methods based on visual vocabularies are an efficient alternative to this type of networks that can be implemented with a reduced number of training samples. Although visual vocabularies have been successfully used in different image classification problems, to our knowledge they have never been tested in the classification of defects in semiconductor material wafers from scanning electron microscopy images. On the rectangular region holding the defect, the scale-invariant feature transform algorithm is applied to determine the points of interest and to calculate the descriptors of the patches containing these points. By training a clustering algorithm with these features, a visual vocabulary is created to describe each image as a bag of visual words. Two approaches for coding the descriptors are studied: (i) The more traditional one performs a count of the occurrence of each visual word, and (ii) a modern approach known as Fisher vector which measures the deviation between each descriptor and the generative Gaussian mixture model. Comparing both variants using support vector machines it is seen that Fisher vector coding is computationally more efficient and reports better classification results. The usefulness of visual vocabularies for defect classification on semiconductor wafers is showed by the 91.89% and 93.13% accuracy obtained for bag of visual words and Fisher vector coding, respectively. The training and testing times are lower than 5 min for both approaches.

Medical Image Registration Algorithm Based on Bounded Generalized Gaussian Mixture Model.

In this paper, a method for medical image registration based on the bounded generalized Gaussian mixture model is proposed. The bounded generalized Gaussian mixture model is used to approach the joint intensity of source medical images. The mixture model is formulated based on a maximum likelihood framework, and is solved by an expectation-maximization algorithm. The registration performance of the proposed approach on different medical images is verified through extensive computer simulations. Empirical findings confirm that the proposed approach is significantly better than other conventional ones.

Multivariate bounded support asymmetric generalized Gaussian mixture model with model selection using minimum message length

In this paper, the bounded support asymmetric generalized Gaussian mixture model (BAGGMM) is proposed for data modeling as an alternative to unbounded mixture models for the cases when the data lies in the bounded support region. The model parameters are learned through maximum likelihood estimation, and Expectation–Maximization (EM) with the Newton Raphson method is adopted for optimization of parameters. Model selection in mixtures is also considered an integral part of clustering; thus, we also have proposed a model selection criterion for BAGGMM through minimum message length. In order to validate the performance of the mixture model, it is applied to image spam detection, object clustering, and visual scene categorization. For the experiments, Spam Hunter, ETHZ, GHIM, and 15-Scene image datasets are adopted, and several clustering scenarios are developed to see the effectiveness of the proposed model. The clustering framework is also compared with AGGMM in a similar setting to all the experiments. In the next phase, the whole clustering framework is extended to examine the performance of the proposed model selection criterion and compared with different techniques to find the optimal mixture component, which further improves the clustering process. The experiments show that the proposed BAGGMM and model selection criterion has demonstrated its success in several learning applications.

Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting, and Regularization

Deep neural networks generalize well despite being exceedingly overparameterized and being trained without explicit regularization. This curious phenomenon has inspired extensive research activity in establishing its statistical principles: Under what conditions is it observed? How do these depend on the data and on the training algorithm? When does regularization benefit generalization? While such questions remain wide open for deep neural nets, recent works have attempted gaining insights by studying simpler, often linear, models. Our paper contributes to this growing line of work by examining binary linear classification under a generative Gaussian mixture model in which the feature vectors take the form ${{\it x}}=\pm{{\eta}}+{{\it q}}$, where for a mean vector $\eta$ and feature noise ${{\it q}} \sim \mathcal{N}(0,{{\Sigma}})$. Motivated by recent results on the implicit bias of gradient descent, we study both max-margin support vector machine (SVM) classifiers (corresponding to logistic loss) and min-norm interpolating classifiers (corresponding to least-squares loss). First, we leverage an idea introduced in [V. Muthukumar et al., arXiv:2005.08054, 2020a] to relate the SVM solution to the min-norm interpolating solution. Second, we derive novel nonasymptotic bounds on the classification error of the latter. Combining the two, we present novel sufficient conditions on the covariance spectrum and on the signal-to-noise ratio (SNR) $SNR={||{{\eta}}||_2^4}/{{\eta}}^T{{\Sigma\eta}}$ under which interpolating estimators achieve asymptotically optimal performance as overparameterization increases. Interestingly, our results extend to a noisy model with constant probability noise flips. Contrary to previously studied discriminative data models, our results emphasize the crucial role of the SNR and its interplay with the data covariance. Finally, via a combination of analytical arguments and numerical demonstrations we identify conditions under which the interpolating estimator performs better than corresponding regularized estimates.

Using the Fisher Vector Approach for Cold Identification

In this paper, we present a computational paralinguistic method for assessing whether a person has an upper respiratory tract infection (i.e. cold) using their speech. Having a system that can accurately assess a cold can be helpful for predicting its propagation. For this purpose, we utilize Mel-frequency Cepstral Coefficients (MFCC) as audio-signal representations, extracted from the utterances, which allowed us to fit a generative Gaussian Mixture Model (GMM) that serves to produce an encoding based on the Fisher Vector (FV) approach. Here, we use the URTIC dataset provided by the organizers of the ComParE Challenge 2017 of the Interspeech Conference. The classification is done by a linear kernel Support Vector Machines (SVM); owing to the high imbalance of classes on the training dataset, we opt for undersampling the majority class, that is, to reduce the number of samples to those of the minority class. We find that applying Power Normalization (PN) and Principal Component Analysis (PCA) on the Fisher vector features is an effective strategy for the classification performance. We get better performance than that of the Bag-of-Audio-Words approach reported in the paper of the challenge.

A Bayesian sampling framework for asymmetric generalized Gaussian mixture models learning

A Bayesian sampling framework for asymmetric generalized Gaussian mixture models learning

Detecting sparse heterogeneous mixtures in a two-sample problem

We consider the problem of detecting sparse heterogeneous mixtures in a two-sample setting from a nonparametric perspective, where the effect manifests itself as a positive shift. We suggest a two-sample higher criticism test, and show that it is first-order comparable to the likelihood ratio test for the generalized Gaussian mixture models in all sparsity regimes.

Multiscale wavelet-based regularized reconstruction algorithm for three-dimensional compressed sensing magnetic resonance imaging

Nowadays, the most dynamic and safe imaging technique used in hospitals to diagnose is magnetic resonance imaging (MRI). In clinical applications, such as follow-up of patients with planning surgery, radiation therapy, therapy response in bone metastases assessment and treatment for brain disorders, repeated MRI scans are performed. Unfortunately, the slow acquisitions of MRI drive cost high by limiting patient throughput as well as limiting the potential indications for use. A mathematical framework called compressed sensing (CS) to accelerate MRI acquisition is used for generating undersampled measurements. Then, the iterative nonlinear numerical method is required for perfect reconstruction from highly undersampled measurements. In this work, the multiscale wavelet domain regularization prior-based iterative reconstruction algorithm is developed to address the above-mentioned reconstruction problem, in which multiscale wavelet domain generalized Gaussian mixture model is utilized for the regularization prior. The regularization-based algorithm is applied to generate three-dimensional (3D) high-quality MRI volume from multichannel k-space measurements in a compressed sensing framework. Several experiments using one synthetic and one real 3D CS multichannel k-space data were used to evaluate and validate the performance of the proposed algorithm. The total variation (TV)-based and soft-thresholding (ST) based-reconstruction methods were implemented to compare with the proposed algorithm. Extensive experimental results demonstrated that the proposed method has improved results in terms of visual quality and quantitative measurements compared to the existing methods.

A General Transfer Learning-based Gaussian Mixture Model for Clustering

Gaussian mixture model (GMM) is a well-known model-based approach for data clustering. However, when the data samples are insufficient, the classical GMM-based clustering algorithms are not effective anymore. Referring to the idea of transfer clustering methods, this paper proposes a general transfer GMM-based clustering framework, which employs the important knowledge extracted from some known source domain to guide and improve the clustering on the target domain with small-scale data. Specifically, three traditional GMM-based clustering approaches are extended to the corresponding transfer clustering versions. Furthermore, to avoid the negative transfer problem, maximum mean discrepancy (MMD) is introduced to search the most matched source domain to provide more positive guidance for data clustering on the target domain. Experiments on synthetic and real-world datasets demonstrate the efficiency of the presented framework compared with several existing transfer clustering algorithms.