Classical Expectation-maximization Algorithm Research Articles

Improving model choice in classification: an approach based on clustering of covariance matrices

This work introduces a refinement of the Parsimonious Model for fitting a Gaussian Mixture. The improvement is based on the consideration of clusters of the involved covariance matrices according to a criterion, such as sharing Principal Directions. This and other similarity criteria that arise from the spectral decomposition of a matrix are the bases of the Parsimonious Model. We show that such groupings of covariance matrices can be achieved through simple modifications of the CEM (Classification Expectation Maximization) algorithm. Our approach leads to propose Gaussian Mixture Models for model-based clustering and discriminant analysis, in which covariance matrices are clustered according to a parsimonious criterion, creating intermediate steps between the fourteen widely known parsimonious models. The added versatility not only allows us to obtain models with fewer parameters for fitting the data, but also provides greater interpretability. We show its usefulness for model-based clustering and discriminant analysis, providing algorithms to find approximate solutions verifying suitable size, shape and orientation constraints, and applying them to both simulation and real data examples.

An Improved Mixture Model of Gaussian Processes and Its Classification Expectation–Maximization Algorithm

The mixture of experts (ME) model is effective for multimodal data in statistics and machine learning. To treat non-stationary probabilistic regression, the mixture of Gaussian processes (MGP) model has been proposed, but it may not perform well in some cases due to the limited ability of each Gaussian process (GP) expert. Although the mixture of Gaussian processes (MGP) and warped Gaussian process (WGP) models are dominant and effective for non-stationary probabilistic regression, they may not be able to handle general non-stationary probabilistic regression in practice. In this paper, we first propose the mixture of warped Gaussian processes (MWGP) model as well as its classification expectation–maximization (CEM) algorithm to address this problem. To overcome the local optimum of the CEM algorithm, we then propose the split and merge CEM (SMC EM) algorithm for MWGP. Experiments were done on synthetic and real-world datasets, which show that our proposed MWGP is more effective than the models used for comparison, and the SMCEM algorithm can solve the local optimum for MWGP.

A Comprehensive Expectation Identification Framework for Multirate Time-Delayed Systems

The expectation maximization (EM) algorithm has been extensively used to solve system identification problems with hidden variables. It needs to calculate a derivative equation and perform a matrix inversion in the EM-M step. The equations related to the EM algorithm may be unsolvable for some complex nonlinear systems, and the matrix inversion has heavy computational costs for large-scale systems. This paper provides two expectation based algorithms with the aim of constructing a comprehensive expectation framework concerning different kinds of time-delayed systems: (1) for a small-scale linear system, the classical EM algorithm can quickly obtain the parameter and time-delay estimates; (2) for a complex nonlinear system with low order, the proposed expectation gradient descent (EGD) algorithm can avoid derivative function calculation; (3) for a large-scale system, the proposed expectation multi-direction (EMD) algorithm does not require eigenvalue calculation and has less computational costs. These two algorithms are developed based on the gradient descent and multi-direction methods. Under such an expectation framework, different kinds of models are identified on a case-by-case basis. The convergence analysis and simulation examples show effectiveness of the algorithms.

Learning hidden Markov models from aggregate observations

In this paper, we propose an algorithm for estimating the parameters of a time-homogeneous hidden Markov model (HMM) from aggregate observations. This problem arises when only the population level counts of the number of individuals at each time step are available, and one seeks to learn the individual HMM from these observations. Our algorithm is built upon the classical expectation–maximization algorithm and the recently proposed aggregate inference algorithm (Sinkhorn belief propagation). We present the parameter learning algorithm for two different settings of HMMs: one with discrete observations and one with continuous observations, and the algorithm exhibits convergence guarantees in both cases. Moreover, our learning framework naturally reduces to the standard Baum–Welch learning algorithm for HMMs when the population size is 1. The efficacy of our algorithm is demonstrated through several numerical experiments.

Comparison of the EM, CEM and SEM algorithms in the estimation of finite mixtures of linear mixed models: a simulation study

Finite mixture models are a widely known method for modelling data that arise from a heterogeneous population. Within the family of mixtures of regression models, mixtures of linear mixed models have also been applied in different areas since, besides taking into consideration the heterogeneity in the population, they also allow to take into account the correlation between observations from the same individual. One of the main issues in mixture models concerns the estimation of the parameters. Maximum likelihood estimation is one of the most used methods in the estimation of the parameters for mixture models. However, the maximization of the log-likelihood function in mixture models is complex, producing in many cases infinite solutions whereby the maximum likelihood estimator may not exist, at least globally. For this reason, it is common to resort to iterative methods, in particular to the Expectation-Maximization (EM) algorithm. However, the slow convergence and the selection of initial values are two of biggest issues of the EM algorithm, the reason why some modified versions of this algorithm have been developed over the years. In this article we compare the performance of the EM, Classification EM (CEM) and Stochastic EM (SEM) algorithms in the estimation of the parameters for mixtures of linear mixed models. In order to evaluate their performance, we carry out a simulation study and a real data application. The results show that the CEM algorithm is the least computationally demanding algorithm, although the three algorithms provide similar maximum likelihood estimates for the parameters.

Supervised clustering of high-dimensional data using regularized mixture modeling.

Identifying relationships between genetic variations and their clinical presentations has been challenged by the heterogeneous causes of a disease. It is imperative to unveil the relationship between the high-dimensional genetic manifestations and the clinical presentations, while taking into account the possible heterogeneity of the study subjects.We proposed a novel supervised clustering algorithm using penalized mixture regression model, called component-wise sparse mixture regression (CSMR), to deal with the challenges in studying the heterogeneous relationships between high-dimensional genetic features and a phenotype. The algorithm was adapted from the classification expectation maximization algorithm, which offers a novel supervised solution to the clustering problem, with substantial improvement on both the computational efficiency and biological interpretability. Experimental evaluation on simulated benchmark datasets demonstrated that the CSMR can accurately identify the subspaces on which subset of features are explanatory to the response variables, and it outperformed the baseline methods. Application of CSMR on a drug sensitivity dataset again demonstrated the superior performance of CSMR over the others, where CSMR is powerful in recapitulating the distinct subgroups hidden in the pool of cell lines with regards to their coping mechanisms to different drugs. CSMR represents a big data analysis tool with the potential to resolve the complexity of translating the clinical representations of the disease to the real causes underpinning it. We believe that it will bring new understanding to the molecular basis of a disease and could be of special relevance in the growing field of personalized medicine.

Unsupervised seismic facies using Gaussian mixture models

As the use of seismic attributes becomes more widespread, multivariate seismic analysis has become more commonplace for seismic facies analysis. Unsupervised machine-learning techniques provide methods of automatically finding patterns in data with minimal user interaction. When using unsupervised machine-learning techniques, such as [Formula: see text]-means or Kohonen self-organizing maps (SOMs), the number of clusters can often be ambiguously defined and there is no measure of how confident the algorithm is in the classification of data vectors. The model-based probabilistic formulation of Gaussian mixture models (GMMs) allows for the number and shape of clusters to be determined in a more objective manner using a Bayesian framework that considers a model’s likelihood and complexity. Furthermore, the development of alternative expectation-maximization (EM) algorithms has allowed GMMs to be more tailored to unsupervised seismic facies analysis. The classification EM algorithm classifies data vectors according to their posterior probabilities that provide a measurement of uncertainty and ambiguity (often called a soft classification). The neighborhood EM (NEM) algorithm allows for spatial correlations to be considered to make classification volumes more realistic by enforcing spatial continuity. Corendering the classification with the uncertainty and ambiguity measurements produces an intuitive map of unsupervised seismic facies. We apply a model-based classification approach using GMMs to a turbidite system in Canterbury Basin, New Zealand, to clarify results from an initial SOM and highlight areas of uncertainty and ambiguity. Special focus on a channel feature in the turbidite system using an NEM algorithm shows it to be more realistic by considering spatial correlations within the data.

IoT service classification and clustering for integration of IoT service platforms

With the rapid development of sensors, wireless communication, and cloud computing, information technology today focuses on service environments created by the Internet of Things (IoT). IoT technologies have become widely used in various contexts including smart homes, building management, surveillance services, and smart farms. Some IoT applications such as Siri are popular in everyday life. IoT requires communication and interaction between various devices and services. To solve the various complex problems associated with IoT services, earlier research focused on IoT service platforms such as gateways and mobile edge computing services. However, the similarities and reusabilities of IoT services have received little attention. In this paper, we develop an IoT service classification and clustering system. We classify the operation of an IoT service into four steps that differ in their characteristics. Based on this classification, we extend the classic EM (expectation–maximization) algorithm to cluster IoT services in terms of their similarities. To validate our proposed classification and clustering system, we divide over 100 commercial IoT services into five clusters, showing that such services are well clustered by similarity and purpose.

A Spherical Hidden Markov Model for Semantics-Rich Human Mobility Modeling

We study the problem of modeling human mobility from semantic trace data, wherein each GPS record in a trace is associated with a text message that describes the user's activity. Existing methods fall short in unveiling human movement regularities for such data, because they either do not model the text data at all or suffer from text sparsity severely. We propose SHMM, a multi-modal spherical hidden Markov model for semantics-rich human mobility modeling. Under the hidden Markov assumption, SHMM models the generation process of a given trace by jointly considering the observed location, time, and text at each step of the trace. The distinguishing characteristic of SHMM is the text modeling part. We use fixed-size vector representations to encode the semantics of the text messages, and model the generation of the l2-normalized text embeddings on a unit sphere with the von Mises-Fisher (vMF) distribution. Compared with other alternatives like multi-variate Gaussian, our choice of the vMF distribution not only incurs much fewer parameters, but also better leverages the discriminative power of text embeddings in a directional metric space. The parameter inference for the vMF distribution is non-trivial since it involves functional inversion of ratios of Bessel functions. We theoretically prove, for the first time, that: 1) the classical Expectation-Maximization algorithm is able to work with vMF distributions; and 2) while closed-form solutions are hard to be obtained for the M-step, Newton's method is guaranteed to converge to the optimal solution with quadratic convergence rate. We have performed extensive experiments on both synthetic and real-life data. The results on synthetic data verify our theoretical analysis; while the results on real-life data demonstrate that SHMM learns meaningful semantics-rich mobility models, outperforms state-of-the-art mobility models for next location prediction, and incurs lower training cost.

Robust Estimation of ARX Models With Time Varying Time Delays Using Variational Bayesian Approach.

This paper is concerned with robust identification of processes with time-varying time delays. In reality, the delay values do not simply change randomly, but there is a correlation between consecutive delays. In this paper, the correlation of time delay is modeled by the transition probability of a Markov chain. Furthermore, the measured data are often contaminated by outliers, and therefore, -distribution is adopted to model the measurement noise. The variational Bayesian (VB) approach is applied to estimate the model parameters along with time delays. Compared with the classical expectation-maximization algorithm, VB approach has the advantage of capturing the uncertainty of the estimated parameter and time delays by providing their full probabilities. The effectiveness of the proposed method is demonstrated by both a numerical example and a pilot-scale hybrid-tank experiment.

Diagonal latent block model for binary data

This paper addresses the problem of co-clustering binary data in the latent block model framework with diagonal constraints for resulting data partitions. We consider the Bernoulli generative mixture model and present three new methods differing in the assumptions made about the degree of homogeneity of diagonal blocks. The proposed models are parsimonious and allow to take into account the structure of a data matrix when reorganizing it into homogeneous diagonal blocks. We derive algorithms for each of the presented models based on the classification expectation-maximization algorithm which maximizes the complete data likelihood. We show that our contribution can outperform other state-of-the-art (co)-clustering methods on synthetic sparse and non-sparse data. We also prove the efficiency of our approach in the context of document clustering, by using real-world benchmark data sets.

Efficient particle-based online smoothing in general hidden Markov models: The PaRIS algorithm

This thesis consists of two papers studying online inference in general hidden Markov models using sequential Monte Carlo methods.The first paper present an novel algorithm, the particle-based, rapid incremental smoother (PaRIS), aimed at efficiently perform online approximation of smoothed expectations of additive state functionals in general hidden Markov models. The algorithm has, under weak assumptions, linear computational complexity and very limited memory requirements. The algorithm is also furnished with a number of convergence results, including a central limit theorem.The second paper focuses on the problem of online estimation of parameters in a general hidden Markov model. The algorithm is based on a forward implementation of the classical expectation-maximization algorithm. The algorithm uses the PaRIS algorithm to achieve an efficient algorithm.

Clustering gene expression data analysis using an improved EM algorithm based on multivariate elliptical contoured mixture models

Clustering gene expression data is an important research topic in bioinformatics because knowing which genes act similarly can lead to the discovery of important biological information. Many clustering algorithms have been used in the field of gene clustering. The multivariate Gaussian mixture distribution function was frequently used as the component of the finite mixture model for clustering, however the clustering cannot be restricted to the normal distribution in the real dataset. In order to make the cluster algorithm strong adaptability, this paper proposes a new scheme for clustering gene expression data based on the multivariate elliptical contoured mixture models (MECMMs). To solve the problem of over-reliance on the initialization, we propose an improved expectation maximization (EM) algorithm by adding and deleting initial value for the classical EM algorithm, and the number of clusters can be treated as a known parameter and inferred with the QAIC criterion. The improved EM algorithm based on the MECMMs is tested and compared with some other clustering algorithms, the performance of our clustering algorithm has been extensively compared over several simulated and real gene expression datasets. Our results indicated that improved EM clustering algorithm is superior to the classical EM algorithm and the support vector machines (SVMs) algorithm, and can be widely used for gene clustering.

Multiway Admixture Deconvolution Using Phased or Unphased Ancestral Panels

We describe a novel method for inferring the local ancestry of admixed individuals from dense genome-wide single nucleotide polymorphism data. The method, called MULTIMIX, allows multiple source populations, models population linkage disequilibrium between markers and is applicable to datasets in which the sample and source populations are either phased or unphased. The model is based upon a hidden Markov model of switches in ancestry between consecutive windows of loci. We model the observed haplotypes within each window using a multivariate normal distribution with parameters estimated from the ancestral panels. We present three methods to fit the model-Markov chain Monte Carlo sampling, the Expectation Maximization algorithm, and a Classification Expectation Maximization algorithm. The performance of our method on individuals simulated to be admixed with European and West African ancestry shows it to be comparable to HAPMIX, the ancestry calls of the two methods agreeing at 99.26% of loci across the three parameter groups. In addition to it being faster than HAPMIX, it is also found to perform well over a range of extent of admixture in a simulation involving three ancestral populations. In an analysis of real data, we estimate the contribution of European, West African and Native American ancestry to each locus in the Mexican samples of HapMap, giving estimates of ancestral proportions that are consistent with those previously reported.

Unsupervised Amplitude and Texture Classification of SAR Images With Multinomial Latent Model

In this paper, we combine amplitude and texture statistics of the synthetic aperture radar images for the purpose of model-based classification. In a finite mixture model, we bring together the Nakagami densities to model the class amplitudes and a 2-D auto-regressive texture model with t-distributed regression error to model the textures of the classes. A non-stationary multinomial logistic latent class label model is used as a mixture density to obtain spatially smooth class segments. The classification expectation-maximization algorithm is performed to estimate the class parameters and to classify the pixels. We resort to integrated classification likelihood criterion to determine the number of classes in the model. We present our results on the classification of the land covers obtained in both supervised and unsupervised cases processing TerraSAR-X, as well as COSMO-SkyMed data.

Automatic segmentation of rodent spinal cord diffusion MR images

MRI, is a key tool for noninvasive spinal cord lesion analysis; however, accurate, quantitative methods for this analysis are lacking. A new, multistep, multidimensional approach, utilizing the classification expectation maximization algorithm, is proposed for MRI segmentation of spinal cord tissues. Diffusion tensor imaging is used to generate multiple images of each spinal slice, with different diffusion direction weightings. The maximum likelihood tissue classifications are then jointly estimated to produce a binary classification image, corresponding to voxels containing either spinal cord or background. Edge detection is employed to find a nonparametric curve encapsulating the entire spinal cord. The algorithm is evaluated using data from in vivo diffusion tensor imaging of control and injured mouse spinal cords. The algorithm is shown to remain accurate for whole spinal cord, white matter, and hemorrhage segmentation in the presence of significant injury. The results of the method are shown to be at least on par with expert manual segmentation.

Fisher scoring: An interpolation family and its Monte Carlo implementations

The Fisher scoring method is widely used for likelihood maximization, but its application can be difficult in situations where the expected information matrix is not available in closed form or when parameters have constraints. In this paper, we describe an interpolation family that generalizes the Fisher scoring method and propose a general Monte Carlo approach that makes these generalized methods also applicable in such situations. With this approach, random samples are generated from the iteratively estimated models and used to provide estimates of the expected information. As a result, the likelihood function can be optimized by repeatedly solving weighted linear regression problems. Specific extensions of this general approach to fitting multivariate normal mixtures and to fitting mixed-effects models with a single discrete random effect are also described. Numerical studies show that the proposed algorithms are fast and reliable to use, as compared with the classical expectation–maximization algorithm.

Fitting mixtures of linear regressions

In most applications, the parameters of a mixture of linear regression models are estimated by maximum likelihood using the expectation maximization (EM) algorithm. In this article, we propose the comparison of three algorithms to compute maximum likelihood estimates of the parameters of these models: the EM algorithm, the classification EM algorithm and the stochastic EM algorithm. The comparison of the three procedures was done through a simulation study of the performance (computational effort, statistical properties of estimators and goodness of fit) of these approaches on simulated data sets. Simulation results show that the choice of the approach depends essentially on the configuration of the true regression lines and the initialization of the algorithms.

Learning Gaussian Mixture Models With Entropy-Based Criteria

In this paper, we address the problem of estimating the parameters of Gaussian mixture models. Although the expectation-maximization (EM) algorithm yields the maximum-likelihood (ML) solution, its sensitivity to the selection of the starting parameters is well-known and it may converge to the boundary of the parameter space. Furthermore, the resulting mixture depends on the number of selected components, but the optimal number of kernels may be unknown beforehand. We introduce the use of the entropy of the probability density function (pdf) associated to each kernel to measure the quality of a given mixture model with a fixed number of kernels. We propose two methods to approximate the entropy of each kernel and a modification of the classical EM algorithm in order to find the optimum number of components of the mixture. Moreover, we use two stopping criteria: a novel global mixture entropy-based criterion called Gaussianity deficiency (GD) and a minimum description length (MDL) principle-based one. Our algorithm, called entropy-based EM (EBEM), starts with a unique kernel and performs only splitting by selecting the worst kernel attending to GD. We have successfully tested it in probability density estimation, pattern classification, and color image segmentation. Experimental results improve the ones of other state-of-the-art model order selection methods.

Bayesian k-Means as a “Maximization-Expectation” Algorithm

We introduce a new class of "maximization-expectation" (ME) algorithms where we maximize over hidden variables but marginalize over random parameters. This reverses the roles of expectation and maximization in the classical expectation-maximization algorithm. In the context of clustering, we argue that these hard assignments open the door to very fast implementations based on data structures such as kd-trees and conga lines. The marginalization over parameters ensures that we retain the ability to infer model structure (i.e., number of clusters). As an important example, we discuss a top-down Bayesian k-means algorithm and a bottom-up agglomerative clustering algorithm. In experiments, we compare these algorithms against a number of alternative algorithms that have recently appeared in the literature.