Expectation Conditional Maximization Algorithm Research Articles

Robust statistical modelling using the multivariate skewtdistribution with complete and incomplete data

Missing data is inevitable in many situations that could hamper data analysis for scientific investigations. We establish flexible analytical tools for multivariate skew t models when fat-tailed, asymmetric and missing observations simultaneously occur in the input data. For the ease of computation and theoretical developments, two auxiliary indicator matrices are incorporated into the model for the determination of observed and missing components of each observation that can effectively reduce the computational complexity. Under the missing at random assumption, we present a Monte Carlo version of the expectation conditional maximization algorithm, which is performed to estimate the parameters and retrieve each missing observation with a single value. Additionally, a Metropolis–Hastings within Gibbs sampler with data augmentation is developed to account for the uncertainty of parameters as well as missing outcomes. The methodology is illustrated through two real data sets.

Joint Decision-Directed Channel and Noise-Variance Estimation for MIMO OFDM/SDMA Systems Based on Expectation-Conditional Maximization

A joint channel impulse response (CIR) and noise-variance estimation scheme is proposed for multiuser multiple-input-multiple-output (MIMO) orthogonal frequency-division multiplexing/space-division multiple access (OFDM/SDMA) systems, which is based on the expectation-conditional maximization (ECM) algorithm. Multiple users communicating over fading channels exhibiting a range of different characteristics are considered in this paper. Channel estimation becomes quite challenging in this scenario since an increased number of independent transmitter-receiver links having different statistical characteristics have to be simultaneously estimated for each subcarrier. To cope with this scenario, we design an ECM-based joint CIR and noise-variance estimator for multiuser MIMO OFDM/SDMA systems, which is capable of simultaneously estimating diverse CIRs and noise variance. Furthermore, we propose a forward error code (FEC)-aided decision-directed channel estimation scheme based on the ECM algorithm, which further improves the ECM algorithm by exploiting the error correction capability of an FEC decoder for iteratively exchanging information between the decoder and the ECM algorithm.

Robust Clustering Using Exponential Power Mixtures

Clustering is a widely used method in extracting useful information from gene expression data, where unknown correlation structures in genes are believed to persist even after normalization. Such correlation structures pose a great challenge on the conventional clustering methods, such as the Gaussian mixture (GM) model, k-means (KM), and partitioning around medoids (PAM), which are not robust against general dependence within data. Here we use the exponential power mixture model to increase the robustness of clustering against general dependence and nonnormality of the data. An expectation-conditional maximization algorithm is developed to calculate the maximum likelihood estimators (MLEs) of the unknown parameters in these mixtures. The Bayesian information criterion is then employed to determine the numbers of components of the mixture. The MLEs are shown to be consistent under sparse dependence. Our numerical results indicate that the proposed procedure outperforms GM, KM, and PAM when there are strong correlations or non-Gaussian components in the data.

Constrained inference in mixed-effects models for longitudinal data with application to hearing loss

In medical studies, endpoints are often measured for each patient longitudinally. The mixed-effects model has been a useful tool for the analysis of such data. There are situations in which the parameters of the model are subject to some restrictions or constraints. For example, in hearing loss studies, we expect hearing to deteriorate with time. This means that hearing thresholds which reflect hearing acuity will, on average, increase over time. Therefore, the regression coefficients associated with the mean effect of time on hearing ability will be constrained. Such constraints should be accounted for in the analysis. We propose maximum likelihood estimation procedures, based on the expectation-conditional maximization either algorithm, to estimate the parameters of the model while accounting for the constraints on them. The proposed methods improve, in terms of mean square error, on the unconstrained estimators. In some settings, the improvement may be substantial. Hypotheses testing procedures that incorporate the constraints are developed. Specifically, likelihood ratio, Wald, and score tests are proposed and investigated. Their empirical significance levels and power are studied using simulations. It is shown that incorporating the constraints improves the mean squared error of the estimates and the power of the tests. These improvements may be substantial. The methodology is used to analyze a hearing loss study.

Robust linear mixed models using the skew t distribution with application to schizophrenia data

We consider an extension of linear mixed models by assuming a multivariate skew t distribution for the random effects and a multivariate t distribution for the error terms. The proposed model provides flexibility in capturing the effects of skewness and heavy tails simultaneously among continuous longitudinal data. We present an efficient alternating expectation-conditional maximization (AECM) algorithm for the computation of maximum likelihood estimates of parameters on the basis of two convenient hierarchical formulations. The techniques for the prediction of random effects and intermittent missing values under this model are also investigated. Our methodologies are illustrated through an application to schizophrenia data.

Model-based classification via mixtures of multivariate [formula omitted]-distributions

A novel model-based classification technique is introduced based on mixtures of multivariate t -distributions. A family of four mixture models is defined by constraining, or not, the covariance matrices and the degrees of freedom to be equal across mixture components. Parameters for each of the resulting four models are estimated using a multicycle expectation–conditional maximization algorithm, where convergence is determined using a criterion based on the Aitken acceleration. A straightforward, but very effective, technique for the initialization of the unknown component memberships is introduced and compared with a popular, more sophisticated, initialization procedure. This novel four-member family is applied to real and simulated data, where it gives good classification performance, even when compared with more established techniques.

Extending mixtures of multivariate t-factor analyzers

Model-based clustering typically involves the development of a family of mixture models and the imposition of these models upon data. The best member of the family is then chosen using some criterion and the associated parameter estimates lead to predicted group memberships, or clusterings. This paper describes the extension of the mixtures of multivariate t-factor analyzers model to include constraints on the degrees of freedom, the factor loadings, and the error variance matrices. The result is a family of six mixture models, including parsimonious models. Parameter estimates for this family of models are derived using an alternating expectation-conditional maximization algorithm and convergence is determined based on Aitken's acceleration. Model selection is carried out using the Bayesian information criterion (BIC) and the integrated completed likelihood (ICL). This novel family of mixture models is then applied to simulated and real data where clustering performance meets or exceeds that of established model-based clustering methods. The simulation studies include a comparison of the BIC and the ICL as model selection techniques for this novel family of models. Application to simulated data with larger dimensionality is also explored.

Fast ML Estimation for the Mixture of Factor Analyzers via an ECM Algorithm

In this brief, we propose a fast expectation conditional maximization (ECM) algorithm for maximum-likelihood (ML) estimation of mixtures of factor analyzers (MFA). Unlike the existing expectation-maximization (EM) algorithms such as the EM in Ghahramani and Hinton, 1996, and the alternating ECM (AECM) in McLachlan and Peel, 2003, where the missing data contains component-indicator vectors as well as latent factors, the missing data in our ECM consists of component-indicator vectors only. The novelty of our algorithm is that closed-form expressions in all conditional maximization (CM) steps are obtained explicitly, instead of resorting to numerical optimization methods. As revealed by experiments, the convergence of our ECM is substantially faster than EM and AECM regardless of whether assessed by central processing unit (CPU) time or number of iterations.

A Computationally Efficient Joint Channel Estimation and Data Detection for SIMO Systems

In this letter, we consider the problem of joint channel estimation and data detection for single-input multi-output (SIMO) systems under fast-fading environment with unknown spatially correlated noise. We present a computationally efficient iterative receiver based on the expectation conditional maximization (ECM) algorithm. Bit error rate (BER) performance of the proposed iterative receiver is simulated and compared with that of the maximum likelihood (ML) receiver with perfect channel state information (CSI). Simulation results show that the proposed iterative receiver achieves nearly the performance of ML receiver with perfect CSI and requires only a few iterations.

Heteroscedastic mixture transition distribution (HMTD) model

The paper further studies the heteroscedastic mixture transition distribution (HMTD) model introduced by Berchtold. Both the expectation and the standard deviation of each component are written as functions of the past of the process. The stationarity conditions are derived. An expectation conditional maximization (ECM) algorithm is used and shown to work well for estimation, the model selection problem is addressed, and the formulaes for computing the observed information matrix are derived. The shape changing feature of conditional distributions makes the model capable of modelling time series with asymmetric or multimodal distribution. The model is applied to several simulated and real datasets with satisfactory results.

Biclustering of Gene Expression Data by an Extension of Mixtures of Factor Analyzers

A challenge in microarray data analysis concerns discovering local structures composed by sets of genes that show homogeneous expression patterns across subsets of conditions. We present an extension of the mixture of factor analyzers model (MFA) allowing for simultaneous clustering of genes and conditions. The proposed model is rather flexible since it models the density of high-dimensional data assuming a mixture of Gaussian distributions with a particular omponent-specific covariance structure. Specifically, a binary and row stochastic matrix representing tissue membership is used to cluster tissues (experimental conditions), whereas the traditional mixture approach is used to define the gene clustering. An alternating expectation conditional maximization (AECM) algorithm is proposed for parameter estimation; experiments on simulated and real data show the efficiency of our method as a general approach to biclustering. The Matlab code of the algorithm is available upon request from authors.

ECM and GLR based multiuser detection with I-CSI

This paper deals with the problem of multiuser detection over a direct-sequence code-division multiple access (DS-CDMA) channel with incomplete channel state informations (I-CSI). We devise and assess two novel recursive detectors based on the expectation conditional maximization (ECM) algorithm and the generalized likelihood ratio (GLR) principle, respectively. Both receivers entail an affordable computational complexity. Moreover, the performance assessment, conducted via Monte Carlo techniques, shows that they achieve satisfactory performance levels and outperform linear detectors.

An EM algorithm for mapping quantitative resistance loci.

Many disease resistance traits in plants have a polygenic background and the disease phenotypes are modified by environmental factors. As a consequence, the phenotypic values usually show a quantitative variation. The phenotypes of such disease traits, however, are often measured in discrete but ordered categories. These traits are called ordinal traits. In terms of disease resistance, they are called quantitative resistance traits, as opposed to qualitative resistance traits, and are controlled by the quantitative resistance loci (QRL). Classical quantitative trait locus mapping methods are not optimal for ordinal trait analysis because the assumption of normal distribution is violated. Methods for mapping binary trait loci are not suitable either because there are more than two categories in ordinal traits. We developed a maximum likelihood method to map these QRL. The method is implemented via a multicycle expectation-conditional-maximization (ECM) algorithm under the threshold model, where we can estimate both the QRL effects and the thresholds that link the disease liability and the categorical phenotype. The method is verified in simulated data under various combinations of the parameters. An SAS program is available to implement the multicycle ECM algorithm. The program can be downloaded from our website at www.statgen.ucr.edu.

Using the EM algorithm to train neural networks: misconceptions and a new algorithm for multiclass classification.

The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perceptron (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets.

Unsupervised robust nonparametric estimation of the hemodynamic response function for any fmri experiment

This paper deals with the estimation of the blood oxygen level-dependent response to a stimulus, as measured in functional magnetic resonance imaging (fMRI) data. A precise estimation is essential for a better understanding of cerebral activations. The most recent works have used a nonparametric framework for this estimation, considering each brain region as a system characterized by its impulse response, the so-called hemodynamic response function (HRF). However, the use of these techniques has remained limited since they are not well-adapted to real fMRI data. Here, we develop a threefold extension to previous works. We consider asynchronous event-related paradigms, account for different trial types and integrate several fMRI sessions into the estimation. These generalizations are simultaneously addressed through a badly conditioned observation model. Bayesian formalism is used to model temporal prior information of the underlying physiological process of the brain hemodynamic response. By this way, the HRF estimate results from a tradeoff between information brought by the data and by our prior knowledge. This tradeoff is modeled with hyperparameters that are set to the maximum-likelihood estimate using an expectation conditional maximization algorithm. The proposed unsupervised approach is validated on both synthetic and real fMRI data, the latter originating from a speech perception experiment.

A Correlated Probit Model for Joint Modeling of Clustered Binary and Continuous Responses

A difficulty in joint modeling of continuous and discrete response variables is the lack of a natural multivariate distribution. For joint modeling of clustered observations on binary and continuous responses, we study a correlated probit model that has an underlying normal latent variable for the binary responses. Catalano and Ryan have factored the model into a marginal and a conditional component and used generalized estimating equations methodology to estimate the effects. We propose a Monte Carlo expectation–conditional maximization algorithm for finding maximum likelihood estimates of the mixed model itself, extending and accelerating an algorithm for models with binary responses. We demonstrate the methodology with a developmental toxicity study measuring fetal weight and a binary malformation status for several litters of mice. A simulation study suggests that efficiency gains of joint fittings over separate fittings of the response variables occur mainly for small datasets with strong correlations between the responses within cluster.

Derivation of a sawtooth iterated extended Kalman smoother via the AECM algorithm

The iterated extended Kalman smoother (IEKS) is derived under expectation-maximization (EM) algorithm formalism, providing insight into the behavior of the suboptimal extended Kalman filter (EKF) and smoother (EKS). Through an investigation of smoothing algorithms that result from variants of the EM algorithm, the sawtooth iterated extended Kalman smoother (SIEKS) and its computationally inexpensive counterparts are proposed via the alternating expectation conditional maximization (AECM) algorithm. The SIEKS is guaranteed to produce a sequence estimate that moves up the likelihood surface. Numerical simulations including frequency tracking examples display the superior performance of the sawtooth EKF over the standard EKF for a range of nonlinear signal models.

An improvement to the interacting multiple model (IMM) algorithm

Computing the optimal conditional mean state estimate for a jump Markov linear system requires exponential complexity, and hence, practical filtering algorithms are necessarily suboptimal. In the target tracking literature, suboptimal multiple-model filtering algorithms, such as the interacting multiple model (IMM) method and generalized pseudo-Bayesian (GPB) schemes, are widely used for state estimation of such systems. We derive a reweighted interacting multiple model algorithm. Although the IMM algorithm is an approximation of the conditional mean state estimator, our algorithm is a recursive implementation of a maximum a posteriori (MAP) state sequence estimator. This MAP estimator is an instance of a previous version of the EM algorithm known as the alternating expectation conditional maximization (AECM) algorithm. Computer simulations indicate that the proposed reweighted IMM algorithm is a competitive alternative to the popular IMM algorithm and GPB methods.

Application of a Mixed Normal Mixture Model for the Estimation of Mastitis-Related Parameters

The current methodology for estimating genetic parameters for SCC (SCS) does not account for the difference in SCS between healthy cows and cows with an intramammary infection (IMI). We propose a two-component finite mixed normal mixture model to estimate IMI prevalence, separate SCS subpopulation means, individual posterior probabilities of IMI, and SCS variance components. The theory is presented and the expectation-conditional maximization algorithm is utilized to compute maximum likelihood estimates. The methodology is illustrated on two simulated data sets based on the current knowledge of SCS parameters. Maximum likelihood estimates of IMI prevalence and SCS subpopulation means were close to simulated values, except for the estimate of IMI prevalence when both subpopulations were almost confounded. Individual posterior probabilities of IMI were always higher among infected than among healthy cows. Error and additive variance components obtained under the mixture model were closer to simulated values than restricted maximum likelihood estimates obtained assuming a homogeneous SCS distribution, especially when subpopulations were completely separated and when mixing proportion was highest. Convergence was linear and rapid when priors were chosen with caution. The advantages of the methodology are demonstrated, and its feasibility for large data sets is discussed.

The EM Algorithm—an Old Folk-song Sung to a Fast New Tune

Summary Celebrating the 20th anniversary of the presentation of the paper by Dempster, Laird and Rubin which popularized the EM algorithm, we investigate, after a brief historical account, strategies that aim to make the EM algorithm converge faster while maintaining its simplicity and stability (e.g. automatic monotone convergence in likelihood). First we introduce the idea of a ‘working parameter’ to facilitate the search for efficient data augmentation schemes and thus fast EM implementations. Second, summarizing various recent extensions of the EM algorithm, we formulate a general alternating expectation–conditional maximization algorithm AECM that couples flexible data augmentation schemes with model reduction schemes to achieve efficient computations. We illustrate these methods using multivariate t-models with known or unknown degrees of freedom and Poisson models for image reconstruction. We show, through both empirical and theoretical evidence, the potential for a dramatic reduction in computational time with little increase in human effort. We also discuss the intrinsic connection between EM-type algorithms and the Gibbs sampler, and the possibility of using the techniques presented here to speed up the latter. The main conclusion of the paper is that, with the help of statistical considerations, it is possible to construct algorithms that are simple, stable and fast.