Mixture Of Factor Analyzers Research Articles

Measurement Error in Earnings Data: Using a Mixture Model Approach to Combine Survey and Register Data

Survey data on earnings tend to contain measurement error. Administrative data are superior in principle, but are worthless in case of a mismatch. We develop methods for prediction in mixture factor analysis models that combine both data sources to arrive at a single earnings figure. We apply the methods to a Swedish data set. Our results show that register earnings data perform poorly if there is a (small) probability of a mismatch. Survey earnings data are more reliable, despite their measurement error. Predictors that combine both and take conditional class probabilities into account outperform all other predictors. This article has supplementary material online.

Extended mixture factor analysis model with covariates for mixed binary and continuous responses

Finite mixture factor analysis provides a parsimonious model to explore latent group structures of high-dimensional data. In this modeling framework, we can explore latent structures for continuous responses. However, dichotomous items are often used to define latent domains in practice. This paper proposes an extended finite mixture factor analysis model with covariates to model mixed continuous and binary responses. We use a Monte Carlo expectation-maximization (MCEM) algorithm to estimate the model. In the E step, closed-form solutions are not available for the conditional expectation of complete data log likelihood, so it is approximated by sample means, which are in turn generated by the Gibbs sampler from the joint conditional distribution of latent variables. To monitor the convergence of the MCEM algorithm, we use bridge sampling to calculate the log likelihood ratio of two successive iterations. We adopt a diagnostic plot of the log likelihood ratio against iterations for monitoring the convergence of the MCEM algorithm. We compare different models based on BIC, in which we approximate the observed data log likelihood by using a Monte Carlo method. We investigate the computational properties of the MCEM algorithm by simulation studies. We use a real data example to illustrate the practical usefulness of the model. Finally, we discuss limitations and possible extensions.

Performance-based control interfaces using mixture of factor analyzers

This paper introduces an approach to performance animation that employs a small number of inertial measurement sensors to create an easy-to-use system for an interactive control of a full-body human character. Our key idea is to construct a global model from a prerecorded motion database and utilize them to construct full-body human motion in a maximum a posteriori framework (MAP). We have demonstrated the effectiveness of our system by controlling a variety of human actions, such as boxing, golf swinging, and table tennis, in real time. One unique property of our system is its ability to learn priors from a large and heterogeneous motion capture database and use them to generate a wide range of natural poses, a capacity that has not been demonstrated in previous data-driven character posing systems.

MLiT: mixtures of Gaussians under linear transformations

The curse of dimensionality hinders the effectiveness of density estimation in high dimensional spaces. Many techniques have been proposed in the past to discover embedded, locally linear manifolds of lower dimensionality, including the mixture of principal component analyzers, the mixture of probabilistic principal component analyzers and the mixture of factor analyzers. In this paper, we propose a novel mixture model for reducing dimensionality based on a linear transformation which is not restricted to be orthogonal nor aligned along the principal directions. For experimental validation, we have used the proposed model for classification of five “hard” data sets and compared its accuracy with that of other popular classifiers. The performance of the proposed method has outperformed that of the mixture of probabilistic principal component analyzers on four out of the five compared data sets with improvements ranging from 0.5 to 3.2%. Moreover, on all data sets, the accuracy achieved by the proposed method outperformed that of the Gaussian mixture model with improvements ranging from 0.2 to 3.4%.

Dimensionally Reduced Model-Based Clustering Through Mixtures of Factor Mixture Analyzers

Dimensionally reduced model-based clustering methods are recently receiving a wide interest in statistics as a tool for performing simultaneously clustering and dimension reduction through one or more latent variables. Among these, Mixtures of Factor Analyzers assume that, within each component, the data are generated according to a factor model, thus reducing the number of parameters on which the covariance matrices depend. In Factor Mixture Analysis clustering is performed through the factors of an ordinary factor analysis which are jointly modelled by a Gaussian mixture. The two approaches differ in genesis, parameterization and consequently clustering performance. In this work we propose a model which extends and combines them. The proposed Mixtures of Factor Mixture Analyzers provide a unified class of dimensionally reduced mixture models which includes the previous ones as special cases and could offer a powerful tool for modelling non-Gaussian latent variables.

Understanding the Underpinnings of the Frontotemporal Lobar Degeneration: Evidence for Benign and Malignant Forms

Frontotemporal lobar degeneration (FTLD) encompasses different clinical subtypes but with overlapping features. Establishing whether FTLD represents a continuum or recognizes distinct subgroups may be crucial for diagnostic purposes and therapeutic approaches. To investigate whether cognitive profiles in a large sample of FTLD patients reflect qualitatively distinct subtypes, variants along a single continuum of severity, or severity differences within subtypes. Latent class analysis (LCA), exploratory factor analysis (FA), and mixture factor analysis (MFA) modeling were applied to a wide neuropsychological assessment data. LCA corresponds to qualitatively distinct subtypes, FA corresponds to quantitatively severity differences, and MFA allows for both subtypes and severity differences within subtypes. The authors consecutively enrolled 314 FTLD patients. A comparison of the different models shows that MFA models provided a superior fit to the data than any of the LCA or exploratory FA models. The "best" MFA model was defined by two continuous variables evaluating the disease severity within two groups of patients ("good" and "bad" performers). These two populations have been called "benign" and "malignant" FTLD. FTLD recognizes distinct subgroups beyond the disease severity, namely a benign form and a more malignant form. This observation needs to be taken into account in future clinical trials and for therapeutic approaches.

The independent factor analysis approach to latent variable modelling

Independent factor analysis (IFA) has recently been proposed in the signal processing literature as a way to model a set of observed variables through linear combinations of latent independent variables and a noise term. A peculiarity of the method is that it defines a probability density function for the latent variables by mixtures of Gaussians. The aim of this paper is to cast the method into a more rigorous statistical framework and to propose some developments. In the first part, we present the IFA model in its population version, address identifiability issues and draw some parallels between the IFA model and the ordinary factor analysis (FA) one. Then we show that the IFA model may be reinterpreted as an independent component analysis-based rotation of an ordinary FA solution. We also give evidence that the IFA model represents a special case of mixture of factor analysers. In the second part, we address inferential issues, also deriving the standard errors for the model parameter estimates and providing model selection criteria. Finally, we present some empirical results on real data sets.

Maximum-likelihood mixture factor analysis model and its application for process monitoring

Abstract In the present paper, a mixture form of the factor analysis model is developed under the maximum-likelihood framework. In this new model structure, different noise levels of process variables have been considered. Afterward, the developed mixture factor analysis model is utilized for process monitoring. To enhance the monitoring performance, a soft combination strategy is then proposed to integrate different local monitoring results into a single monitoring chart, which is based on the Bayesian inference method. To test the modeling and monitoring performance of the proposed mixture factor analysis method, a numerical example and the Tennessee Eastman (TE) benchmark case studies are provided.

New statistical model for radar HRRP target recognition

To resolve the problem of how to determine the proper number of the mixture models for radar high-resolution range profile (HRRP) target recognition. This paper develops a variational Bayesian mixture of factor analyzers (VBMFA) model. This method can automatically determine the optimal number of models by birth-death moves and can accurately describe the statistical characteristics of HRRP. So the VBMFA method should have better recognition performance than factor analysis and mixtures of factor analyzers method, and experimental results for measured data proved this conclusion.

Managing polysemy and synonymy in science mapping using the mixtures of factor analyzers model

AbstractA new method for mapping the semantic structure of science is described. We assume that different researchers, working on the same set of research problems, will use the same words for concepts central to their research problems. Therefore, different research fields and disciplines should be identifiable by different words and the pattern of co‐occurring words. In natural language, however, there is quite some diversity because many words have multiple meaning. In addition, the same meaning can be expressed by using different words. We argue that traditional factor analytic and cluster analytic techniques are inadequate for mapping the semantic structure if such polysemous and synonymous words are present. Instead, an alternative model, the mixtures of factor analyzers (MFA) model, is utilized. This model extends the traditional factor analytic model by allowing multiple centroids of the dataset. We argue that this model is structurally better suited to map the semantic structure of science. The model is illustrated by a case study of the uncertainty literature sampled from data from the ISI Web of Science. The MFA model is applied with the goal of discovering multiple, potentially incommensurate, conceptualizations of uncertainty in the literature. In this way, the MFA model can help in creating understanding of the use of language in science, which can benefit multidisciplinary research and interdisciplinary understanding, and assist in the development of multidisciplinary taxonomies of science.

Modelling transcriptional regulation with a mixture of factor analyzers and variational Bayesian expectation maximization.

Understanding the mechanisms of gene transcriptional regulation through analysis of high-throughput postgenomic data is one of the central problems of computational systems biology. Various approaches have been proposed, but most of them fail to address at least one of the following objectives: (1) allow for the fact that transcription factors are potentially subject to posttranscriptional regulation; (2) allow for the fact that transcription factors cooperate as a functional complex in regulating gene expression, and (3) provide a model and a learning algorithm with manageable computational complexity. The objective of the present study is to propose and test a method that addresses these three issues. The model we employ is a mixture of factor analyzers, in which the latent variables correspond to different transcription factors, grouped into complexes or modules. We pursue inference in a Bayesian framework, using the Variational Bayesian Expectation Maximization (VBEM) algorithm for approximate inference of the posterior distributions of the model parameters, and estimation of a lower bound on the marginal likelihood for model selection. We have evaluated the performance of the proposed method on three criteria: activity profile reconstruction, gene clustering, and network inference.

Modelling Transcriptional Regulation with a Mixture of Factor Analyzers and Variational Bayesian Expectation Maximization

Modelling Transcriptional Regulation with a Mixture of Factor Analyzers and Variational Bayesian Expectation Maximization

Fast ML Estimation for the Mixture of Factor Analyzers via an ECM Algorithm

In this brief, we propose a fast expectation conditional maximization (ECM) algorithm for maximum-likelihood (ML) estimation of mixtures of factor analyzers (MFA). Unlike the existing expectation-maximization (EM) algorithms such as the EM in Ghahramani and Hinton, 1996, and the alternating ECM (AECM) in McLachlan and Peel, 2003, where the missing data contains component-indicator vectors as well as latent factors, the missing data in our ECM consists of component-indicator vectors only. The novelty of our algorithm is that closed-form expressions in all conditional maximization (CM) steps are obtained explicitly, instead of resorting to numerical optimization methods. As revealed by experiments, the convergence of our ECM is substantially faster than EM and AECM regardless of whether assessed by central processing unit (CPU) time or number of iterations.

Radar HRRP Statistical Recognition: Parametric Model and Model Selection

Statistical modeling for radar high-resolution range profile (HRRP) is a challenging task in radar HRRP statistical recognition. Theoretical analysis and experimental results show that elements in an HRRP sample are statistically correlated and non-Gaussian distributed. First, this paper introduces three joint-Gaussian models, i.e., subspace approximation model, probability principal components analysis (PPCA) model and factor analysis (FA) model, into radar HRRP statistical recognition. Due to the experimental results, we can have the conclusion that the jointly non-Gaussian distributed HRRP samples approximately follow the joint-Gaussian distribution described by FA model. Therefore, we can apply FA model to radar HRRP statistical recognition rather than a joint-Gaussian mixture model, e.g., PPCA mixture model or FA mixture model, which is a more accurate choice for modeling non-Gaussian distributed correlations in multidimensional data but with high learning complexity and large computation burden, and the difficulty in the statistical modeling for HRRP samples is largely reduced. Second, this paper concerns model selection of FA model in radar HRRP statistical recognition, in which there are two issues, i.e., the partition of target-aspect frames and the determination of the number of factors in each frame. Based on the Akaike information criterion (AIC) and the Bayes' information criterion (BIC), an iterated algorithm for model selection is proposed in this paper, which can automatically give the optimal aspect-frame boundaries and determine the optimal number of factors in each aspect-frame. The recognition experiments based on measured data show that the proposed adaptive partition approach can further improve the recognition performance with higher recognition efficiency.

Robust Cluster Analysis via Mixture Models

Finite mixture models are being increasingly used to model the distributions of a wide variety of random phenomena and to cluster data sets. In this paper, we focus on the use of normal mixture models to cluster data sets of continuous multivariate data. As normality based methods of estimation are not robust, we review the use of t component distributions. With the t mixture model-based approach, the normal distribution for each component in the mixture model is embedded in a wider class of elliptically symmetric distributions with an additional parameter called the degrees of freedom. The advantage of the t mixture model is that, although the number of outliers needed for breakdown is almost the same as with the normal mixture model, the outliers have to be much larger. We also consider the use of the t distribution for the robust clustering of high-dimensional data via mixtures of factor analyzers. The latter enable a mixture model to be fitted to data which have high dimension relative to the number of data points to be clustered.

Classification of microarray data with factor mixture models

The classification of few tissue samples on a very large number of genes represents a non-standard problem in statistics but a usual one in microarray expression data analysis. In fact, the dimension of the feature space (the number of genes) is typically much greater than the number of tissues. We consider high-density oligonucleotide microarray data, where the expression level is associated to an 'absolute call', which represents a qualitative indication of whether or not a transcript is detected within a sample. The 'absolute call' is generally not taken in consideration in analyses. In contrast to frequently used cluster analysis methods to analyze gene expression data, we consider a problem of classification of tissues and of the variables selection. We adopted methodologies formulated by Ghahramani and Hinton and Rocci and Vichi for simultaneous dimensional reduction of genes and classification of tissues; trying to identify genes (denominated 'markers') that are able to distinguish between two known different classes of tissue samples. In this respect, we propose a generalization of the approach proposed by McLachlan et al. by advising to estimate the distribution of log LR statistic for testing one versus two component hypothesis in the mixture model for each gene considered individually, using a parametric bootstrap approach. We compare conditional (on 'absolute call') and unconditional analyses performed on dataset described in Golub et al. We show that the proposed techniques improve the results of classification of tissue samples with respect to known results on the same benchmark dataset. The software of Ghahramani and Hinton is written in Matlab and available in 'Mixture of Factor Analyzers' on http://www.gatsby.ucl.ac.uk/~zoubin/software.html while the software of Rocci and Vichi is available upon request from the authors.

Latent Classification Models

One of the simplest, and yet most consistently well-performing set of classifiers is the Naïve Bayes models. These models rely on two assumptions: (i) All the attributes used to describe an instance are conditionally independent given the class of that instance, and (ii) all attributes follow a specific parametric family of distributions. In this paper we propose a new set of models for classification in continuous domains, termed latent classification models. The latent classification model can roughly be seen as combining the Naïve Bayes model with a mixture of factor analyzers, thereby relaxing the assumptions of the Naïve Bayes classifier. In the proposed model the continuous attributes are described by a mixture of multivariate Gaussians, where the conditional dependencies among the attributes are encoded using latent variables. We present algorithms for learning both the parameters and the structure of a latent classification model, and we demonstrate empirically that the accuracy of the proposed model is significantly higher than the accuracy of other probabilistic classifiers.

Parameter Sharing in Mixture of Factor Analyzers for Speaker Identification

This paper investigates the parameter tying structures of a mixture of factor analyzers (MFA) and discriminative training of MFA for speaker identification. The parameters of factor loading matrices or diagonal matrices are shared in different mixtures of MFA. Then, minimum classification error (MCE) training is applied to the MFA parameters to enhance the discrimination ability. The result of a text-independent speaker identification experiment shows that MFA outperforms the conventional Gaussian mixture model (GMM) with diagonal or full covariance matrices and achieves the best performance when sharing the diagonal matrices, resulting in a relative gain of 26% over the GMM with diagonal covariance matrices. The improvement is more significant especially in sparse training data condition. The recognition performance is further improved by MCE training with an additional gain of 3% error reduction.

Mixtures of factor analyzers: an extension with covariates

This paper examines the analysis of an extended finite mixture of factor analyzers (MFA) where both the continuous latent variable (common factor) and the categorical latent variable (component label) are assumed to be influenced by the effects of fixed observed covariates. A polytomous logistic regression model is used to link the categorical latent variable to its corresponding covariate, while a traditional linear model with normal noise is used to model the effect of the covariate on the continuous latent variable. The proposed model turns out be in various ways an extension of many existing related models, and as such offers the potential to address some of the issues not fully handled by those previous models. A detailed derivation of an EM algorithm is proposed for parameter estimation, and latent variable estimates are obtained as by-products of the overall estimation procedure.

Removal of artifacts and fluctuations from MEG data by clustering methods

We proposed to cluster a set of magnetoencephalogram (MEG) records using mixture of factor analyzers to remove those outlying records that were contaminated with artifacts. We showed the effectiveness of the proposed clustering approach by applying it to visual and auditory evoked MEG data.