Generalized EM Algorithm Research Articles

Discriminative Topic Modeling Based on Manifold Learning

Topic modeling has become a popular method used for data analysis in various domains including text documents. Previous topic model approaches, such as probabilistic Latent Semantic Analysis (pLSA) and Latent Dirichlet Allocation (LDA), have shown impressive success in discovering low-rank hidden structures for modeling text documents. These approaches, however do not take into account the manifold structure of the data, which is generally informative for nonlinear dimensionality reduction mapping. More recent topic model approaches, Laplacian PLSI (LapPLSI) and Locally-consistent Topic Model (LTM), have incorporated the local manifold structure into topic models and have shown resulting benefits. But they fall short of achieving full discriminating power of manifold learning as they only enhance the proximity between the low-rank representations of neighboring pairs without any consideration for non-neighboring pairs. In this article, we propose a new approach, Discriminative Topic Model (DTM), which separates non-neighboring pairs from each other in addition to bringing neighboring pairs closer together, thereby preserving the global manifold structure as well as improving local consistency. We also present a novel model-fitting algorithm based on the generalized EM algorithm and the concept of Pareto improvement. We empirically demonstrate the success of DTM in terms of unsupervised clustering and semisupervised classification accuracies on text corpora and robustness to parameters compared to state-of-the-art techniques.

A skew-normal factor model for the analysis of student satisfaction towards university courses

Classical factor analysis relies on the assumption of normally distributed factors that guarantees the model to be estimated via the maximum likelihood method. Even when the assumption of Gaussian factors is not explicitly formulated and estimation is performed via the iterated principal factors’ method, the interest is actually mainly focussed on the linear structure of the data, since only moments up to the second ones are involved. In many real situations, the factors could not be adequately described by the first two moments only. For example, skewness characterizing most latent variables in social analysis can be properly measured by the third moment: the factors are not normally distributed and covariance is no longer a sufficient statistic. In this work we propose a factor model characterized by skew-normally distributed factors. Skew-normal refers to a parametric class of probability distributions, that extends the normal distribution by an additional shape parameter regulating the skewness. The model estimation can be solved by the generalized EM algorithm, in which the iterative Newthon–Raphson procedure is needed in the M-step to estimate the factor shape parameter. The proposed skew-normal factor analysis is applied to the study of student satisfaction towards university courses, in order to identify the factors representing different aspects of the latent overall satisfaction.

An Examplar-Based Approach for Texture Compaction Synthesis and Retrieval

A texture representation should corroborate various functions of a texture. In this paper, we present a novel approach that incorporates texture features for retrieval in an examplar-based texture compaction and synthesis algorithm. The original texture is compacted and compressed in the encoder to obtain a thumbnail texture, which the decoder then synthesizes to obtain a perceptually high quality texture. We propose using a probabilistic framework based on the generalized EM algorithm to analyze the solutions of the approach. Our experiment results show that a high quality synthesized texture can be generated in the decoder from a compressed thumbnail texture. The number of bits in the compressed thumbnail is 400 times lower than that in the original texture and 50 times lower than that needed to compress the original texture using JPEG2000. We also show that, in terms of retrieval and synthesization, our compressed and compacted textures perform better than compressed cropped textures and compressed compacted textures derived by the patchwork algorithm.

A <formula formulatype="inline"><tex>$q$</tex></formula>-Parameterized Deterministic Annealing EM Algorithm Based on Nonextensive Statistical Mechanics

We propose a q-parameterized deterministic annealing expectation maximization (q-DAEM) algorithm for parameter estimation motivated by the concept of Tsallis entropy that originates from the nonextensive statistical mechanics. The q-DAEM algorithm combines the feature of annealing algorithms to reduce initialization sensitivity and that of q-EM algorithms to achieve fast convergence. The q-EM algorithm is a one-parameter generalized EM algorithm that has been previously proposed by the authors. By interpreting the EM algorithm via likelihood lower bound maximization, we build the fundamental interconnections among DAEM, q-DAEM, and statistical mechanics. To illustrate the benefits of using q-DAEM, we investigate two applications: finite mixture model estimation in data clustering; and joint channel estimation and data detection in communication systems, where we show that the q-DAEM algorithm achieves superior performance over the EM algorithm for both applications.

A structured variational learning approach for switching latent factor models

A data-driven approach for modeling volatility dynamics and co-movements in financial markets is introduced. Special emphasis is given to multivariate conditionally heteroscedastic factor models in which the volatilities of the latent factors depend on their past values, and the parameters are driven by regime switching in a latent state variable. We propose an innovative indirect estimation method based on the generalized EM algorithm principle combined with a structured variational approach that can handle models with large cross-sectional dimensions. Extensive Monte Carlo simulations and preliminary experiments with financial data show promising results.

Low complexity generalized EM algorithm for blind channel estimation and data detection in optical communication systems

Expectation–Maximization algorithm (EM) has been used in the past for blind estimation of intersymbol interference channels characterized by additive white Gaussian noise. When the channel is characterized by non-Gaussian, signal-dependent noise, the computational complexity of direct application of EM becomes prohibitively high. In this paper, a low complexity generalized EM algorithm is presented. The proposed algorithm achieves a major reduction in computational complexity compared to the EM algorithm and can be applied to nonlinear finite memory channels with non-Gaussian signal-dependent noise. Simulation results are presented for intensity modulated direct detection optical channel that is characterized by non-central chi-square distribution noise.

An EM algorithm for the block mixture model

Although many clustering procedures aim to construct an optimal partition of objects or, sometimes, of variables, there are other methods, called block clustering methods, which consider simultaneously the two sets and organize the data into homogeneous blocks. Recently, we have proposed a new mixture model called block mixture model which takes into account this situation. This model allows one to embed simultaneous clustering of objects and variables in a mixture approach. We have studied this probabilistic model under the classification likelihood approach and developed a new algorithm for simultaneous partitioning based on the Classification EM algorithm. In this paper, we consider the block clustering problem under the maximum likelihood approach and the goal of our contribution is to estimate the parameters of this model. Unfortunately, the application of the EM algorithm for the block mixture model cannot be made directly; difficulties arise due to the dependence structure in the model and approximations are required. Using a variational approximation, we propose a generalized EM algorithm to estimate the parameters of the block mixture model and, to illustrate our approach, we study the case of binary data by using a Bernoulli block mixture.

Modeling Inverse Covariance Matricesby Basis Expansion

This paper proposes a new covariance modeling technique for Gaussian mixture models. Specifically the inverse covariance (precision) matrix of each Gaussian is expanded in a rank-1 basis i.e., /spl Sigma//sub j//sup -1/=P/sub j/=/spl Sigma//sub k=1//sup D//spl lambda//sub k//sup j/a/sub k/a/sub k//sup T/, /spl lambda//sub k//sup j//spl isin//spl Ropf/,a/sub k//spl isin//spl Ropf//sup d/. A generalized EM algorithm is proposed to obtain maximum likelihood parameter estimates for the basis set {a/sub k/a/sub k//sup T/}/sub k=1//sup D/ and the expansion coefficients {/spl lambda//sub k//sup j/}. This model, called the extended maximum likelihood linear transform (EMLLT) model, is extremely flexible: by varying the number of basis elements from D=d to D=d(d+1)/2 one gradually moves from a maximum likelihood linear transform (MLLT) model to a full-covariance model. Experimental results on two speech recognition tasks show that the EMLLT model can give relative gains of up to 35% in the word error rate over a standard diagonal covariance model, 30% over a standard MLLT model.

Target-Directed Mixture Dynamic Models for Spontaneous Speech Recognition

In this paper, a novel mixture linear dynamic model (MLDM) for speech recognition is developed and evaluated, where several linear dynamic models are combined (mixed) to represent different vocal-tract-resonance (VTR) dynamic behaviors and the mapping relationships between the VTRs and the acoustic observations. Each linear dynamic model is formulated as the state-space equations, where the VTRs target-directed property is incorporated in the state equation and a linear regression function is used for the observation equation that approximates the nonlinear mapping relationship. A version of the generalized EM algorithm is developed for learning the model parameters, where the constraint that the VTR targets change at the segmental level (rather than at the frame level) is imposed in the parameter learning and model scoring algorithms. Speech recognition experiments are carried out to evaluate the new model using the N-best re-scoring paradigm in a Switchboard task. Compared with a baseline recognizer using the triphone HMM acoustic model, the new recognizer demonstrated improved performance under several experimental conditions. The performance was shown to increase with an increased number of the mixture components in the model.

Least absolute deviations estimation via the EM algorithm

This paper derives EM and generalized EM (GEM) algorithms for calculating least absolute deviations (LAD) estimates of the parameters of linear and nonlinear regression models. It shows that Schlossmacher's iterative reweighted least squares algorithm for calculating LAD estimates (E.J. Schlossmacher, Journal of the American Statistical Association 68: 857–859, 1973) is an EM algorithm. A GEM algorithm for computing LAD estimates of the parameters of nonlinear regression models is also provided and is applied in some examples.

A Note on parameter and standard error estimation in adaptive robust regression

This paper introduces practical methods of parameter and standard error estimation for adaptive robust regression where errors are assumed to be from a normal/independent family of distributions. In particular, generalized EM algorithms (GEM) are considered for the two cases of t and slash families of distributions. For the t family, a one step method is proposed to estimate the degree of freedom parameter. Use of empirical information is suggested for standard error estimation. It is shown that this choice leads to standard errors that can be obtained as a by-product of the GEM algorithm. The proposed methods, as discussed, can be implemented in most available nonlinear regression programs. Details of implementation in SAS NLIN are given using two specific examples.

Spontaneous speech recognition using a statistical coarticulatory model for the vocal-tract-resonance dynamics.

A statistical coarticulatory model is presented for spontaneous speech recognition, where knowledge of the dynamic, target-directed behavior in the vocal tract resonance is incorporated into the model design, training, and in likelihood computation. The principal advantage of the new model over the conventional HMM is the use of a compact, internal structure that parsimoniously represents long-span context dependence in the observable domain of speech acoustics without using additional, context-dependent model parameters. The new model is formulated mathematically as a constrained, nonstationary, and nonlinear dynamic system, for which a version of the generalized EM algorithm is developed and implemented for automatically learning the compact set of model parameters. A series of experiments for speech recognition and model synthesis using spontaneous speech data from the Switchboard corpus are reported. The promise of the new model is demonstrated by showing its consistently superior performance over a state-of-the-art benchmark HMM system under controlled experimental conditions. Experiments on model synthesis and analysis shed insight into the mechanism underlying such superiority in terms of the target-directed behavior and of the long-span context-dependence property, both inherent in the designed structure of the new dynamic model of speech.

Nonlinear compensation for stochastic matching

The performance of an automatic speech recognizer degrades when there exists an acoustic mismatch between the training and the testing conditions in the data. Though it is certain that the mismatch is nonlinear, its exact form is unknown. Tackling the problem of nonlinear mismatches is a difficult task that has not been adequately addressed before. We develop an approach that uses nonlinear transformations in the stochastic matching framework to compensate for acoustic mismatches. The functional form of the nonlinear transformation is modeled by neural networks. We develop a new technique to train neural networks using the generalized EM algorithm. This technique eliminates the need for stereo databases, which are difficult to obtain in practical applications. The new technique is data-driven and hence can be used under a wide variety of conditions without a priori knowledge of the environment. Using this technique, we show that we can provide improvement under various types of acoustic mismatch; in some cases a 72% reduction in word error rate is achieved.

Estimation of general linear-bilinear models for two-way tables

We define the General Linear-Bilinear Model (GLBM) for data arranged as a r×c table as . This includes linear-bilinear models known as Additive Main Effects and Multiplicative Interaction, Rows Regression, Columns Regression, and Shifted Multiplicative models as special cases, but further allows for inclusion of regression on covariates as additional linear terms and for estimation of missing cells. A GLBM is defined as “balanced” if least squares estimates of its linear effects are free of the bilinear effects. A closed form least squares solution exists if the GLBM is balanced or if and is of rank one for all k, where q is the number of linear effects fitted within each (and every) row. In all GLBMs, the least squares estimates of the multiplicative terms are obtained by singular value decomposition of the matrix A of deviations but, if the GLBM is unbalanced, solutions for the depend on the decomposition to be obtained. For such cases, iterative Newton-Raphson and generalized EM algorithms are developed. Closed form solutions for unbalanced GLBMs with and all rank(X k) = 1 can be exploited for finding initial values for iterative solutions for smaller t, as well as. for models with some rank(X k) > 1. An example is presented in which, within each level of the column factor, there is regression on a covariate and adjustment for incomplete blocking.

Solution of the basic equation of stellar statistics

The basic equation of stellar statistics connects the probability density function of a measurable quantity with the probability density of two variables, which cannot be observed directly, by the Bayes theorem of conditional probabilities. The resulting relation is a Fredholm-type integral equation of the first kind. If the two background variables are statistically independent we recover the convolution equation. The analytical solution based on the Fourier transformation is very sensitive to high-frequency noise. Eddington's solution attempts to find the unknown function in form of a series Sigma gamma jh(i)(z). Malmquist's method computes the conditional probability of the unknown variable assuming that the observed variable is given. The statistical aspect of the problem is expressed if one uses Lucy's algorithm which is a particular form of the more general EM algorithm. Dolan's matrix method solves numerically the matrix equation which approximates the integral equation. Methods are superior which retain the true statistical nature of the problem.

A statistical model for formant-transition microsegments of speech incorporating locus equations

Abstract Locus equations, which describe linear relationships between the onset and steady-state formant values in consonant-vowel syllables, have recently been measured using a large quantity of acoustic data and have been proposed as a source of relational invariance for stop place categorization (H. Sussman et al., 1991). In this paper we present a statistical model which utilizes the conceptualization of the locus equations as a basis for parametric of modeling of phonetic contexts — place of articulation, and of their acoustic consequences — formant transitions. The model is based on a hidden Markov model representation of formant-transition microsegments of speech. We develop a generalized EM algorithm for automatic estimation of the model parameters. The proposed model is capable of generalizing consonant characteristics from a small training data set where the contextual information is only sparsely represented, and is hence applicable to solving very large vocabulary speech recognition problems.

A classification EM algorithm for clustering and two stochastic versions

Setting the optimization-based clustering methods under the classification maximum likelihood approach, we define and study a general Classification EM algorithm. Then, we derive from this algorithm two stochastic algorithms, incorporating random perturbations, to reduce the initial-position dependence of the classical optimization clustering algorithms. Numerical experiments, reported for the variance criterion, show that both stochastic algorithms perform well compared with the standard k-means algorithm which is a particular version of the Classification EM algorithm.

A generalized EM algorithm for 3-D Bayesian reconstruction from Poisson data using Gibbs priors

A generalized expectation-maximization (GEM) algorithm is developed for Bayesian reconstruction, based on locally correlated Markov random-field priors in the form of Gibbs functions and on the Poisson data model. For the M-step of the algorithm, a form of coordinate gradient ascent is derived. The algorithm reduces to the EM maximum-likelihood algorithm as the Markov random-field prior tends towards a uniform distribution. Three different Gibbs function priors are examined. Reconstructions of 3-D images obtained from the Poisson model of single-photon-emission computed tomography are presented.

Analyzing Marketing Research Data with Incomplete Information on the Dependent Variable

The author presents the general EM algorithm for analyzing incomplete data. As a specific application of the EM algorithm, a model is proposed for analyzing incomplete data in an important class of problems in marketing research. A simple estimation procedure also is developed. The model is investigated through Monté Carlo studies as well as empirically with encouraging results. Some of the advantages and limitations of the approach are discussed.

Maximum Likelihood Solution to the Parafac Model

The traditional factor analytic view of the PARAFAC model and its extension to a four mode situation with the derivation of the maximum likelihood estimation procedure by the generalized EM algorithm was presented. The four mode model was applied to six ASVAB data matrices defined by three specialty (clerical, mechanical, and electrical), times two services, (Air Force and Marine Corps) and successfully recovered the usual four dimensional structure without any rotation. The specialty and service differences was expressed in terms of different weighting of the common factor structure. A model which allows us to compare the factor score means was also investigated.