Mixture Of Factor Analyzers Research Articles

Detecting non-content-based response styles in survey data: An application of mixture factor analysis

It is common for some participants in self-report surveys to be careless, inattentive, or lacking in effort. Data quality can be severely compromised by responses that are not based on item content (non-content-based [nCB] responses), leading to strong biases in the results of data analysis and misinterpretation of individual scores. In this study, we propose a specification of factor mixture analysis (FMA) to detect nCB responses. We investigated the usefulness and effectiveness of the FMA model in detecting nCB responses using both simulated data (Study 1) and real data (Study 2). In the first study, FMA showed reasonably robust sensitivity (.60 to .86) and excellent specificity (.96 to .99) on mixed-worded scales, suggesting that FMA had superior properties as a screening tool under different sample conditions. However, FMA performance was poor on scales composed of only positive items because of the difficulty in distinguishing acquiescent patterns from valid responses representing high levels of the trait. In Study 2 (real data), FMA detected a minority of cases (6.5%) with highly anomalous response patterns. Removing these cases resulted in a large increase in the fit of the unidimensional model and a substantial reduction in spurious multidimensionality.

Using Mixture Factor Analysis to Counter Faking

"Using Mixture Factor Analysis to Counter Faking." Multivariate Behavioral Research, ahead-of-print(ahead-of-print), pp. 1–2

Variational inference and sparsity in high-dimensional deep Gaussian mixture models

Gaussian mixture models are a popular tool for model-based clustering, and mixtures of factor analyzers are Gaussian mixture models having parsimonious factor covariance structure for mixture components. There are several recent extensions of mixture of factor analyzers to deep mixtures, where the Gaussian model for the latent factors is replaced by a mixture of factor analyzers. This construction can be iterated to obtain a model with many layers. These deep models are challenging to fit, and we consider Bayesian inference using sparsity priors to further regularize the estimation. A scalable natural gradient variational inference algorithm is developed for fitting the model, and we suggest computationally efficient approaches to the architecture choice using overfitted mixtures where unnecessary components drop out in the estimation. In a number of simulated and two real examples, we demonstrate the versatility of our approach for high-dimensional problems, and demonstrate that the use of sparsity inducing priors can be helpful for obtaining improved clustering results.

A fisher score-based multi-instance learning method assisted by mixture of factor analysis

Multi-instance learning (MIL) is a problem with significant importance due to the existence of ambiguous labels in many practical tasks. The training data in MIL problem is represented by labeled bags containing a various number of unlabeled instances. This inexact label information has brought complexities to the classification of instances and bags. Fisher scores have been utilized to extract fixed-length representation vectors from the MIL bags with the assistance of generative models. However, the generative model utilized by the existing methods might disregard the covariance information. In this paper, we propose a multi-instance learning method (miMFA) utilizing Fisher scores derived from the mixture of factor analysis (MFA) model. The MFA generates nonlinear data with a set of local factor analysis models, while each local model approximates the full covariance Gaussian using latent factors. Thus, the MFA could cover the data distribution and generate Fisher scores effectively. The MFA-based Fisher score is then utilized to form the bag representation. Moreover, a reinforcement vector and the complementary information from auxiliary MIL methods are introduced for performance refinement. The proposed miMFA is evaluated on 11 MIL datasets, and achieves satisfactory results compared with the existing methods.

On the identifiability of Bayesian factor analytic models

A well known identifiability issue in factor analytic models is the invariance with respect to orthogonal transformations. This problem burdens the inference under a Bayesian setup, where Markov chain Monte Carlo (MCMC) methods are used to generate samples from the posterior distribution. We introduce a post-processing scheme in order to deal with rotation, sign and permutation invariance of the MCMC sample. The exact version of the contributed algorithm requires to solve 2^q assignment problems per (retained) MCMC iteration, where q denotes the number of factors of the fitted model. For large numbers of factors two approximate schemes based on simulated annealing are also discussed. We demonstrate that the proposed method leads to interpretable posterior distributions using synthetic and publicly available data from typical factor analytic models as well as mixtures of factor analyzers. An R package is available online at CRAN web-page.

Missing data imputation using mixture factor analysis for building electric load data

We propose a mixture factor analysis (MFA) method for estimating missing values in building electric load data. Buildings consume a tremendous amount of energy. Thanks to the recent advances in data technologies such as machine learning and applied statistics, data-driven approaches to making buildings more energy-efficient become a major research area. However, building electric load data suffer from quality issues due to data missing originated from malfunctioning sensors, network stability, and other environmental causes. We note that data missing can occur even under advanced Internet Technology (IT) systems such as energy information systems (EIS) and energy management systems (EMS) due to signal stability and low-speed computers. The existence of missing data may significantly affect building operations, causing inaccuracy in evaluating building status and forecasting future electric demands. In this respect, dealing with missing data problems should be as important as developing highly accurate forecasting algorithms. While investigating load data, we find that building electric loads exhibit distinct patterns with cyclic rotations that we can take advantage of in both model design and selection stages. Motivated by the finding, unlike the previous studies designed for general time-series data, we propose a novel data imputation model to represent patterns and their cyclic rotations in electric load data. Simulation studies reveal that the proposed model works well when the time window size is a divisor of the cycle length, which significantly reduces model selection efforts. Numerical results with two real data sets justify our findings and the performance of the proposed approach against benchmark methods.

Evaluating Covariate Effects on ESM Measurement Model Changes with Latent Markov Factor Analysis: A Three-Step Approach

Invariance of the measurement model (MM) between subjects and within subjects over time is a prerequisite for drawing valid inferences when studying dynamics of psychological factors in intensive longitudinal data. To conveniently evaluate this invariance, latent Markov factor analysis (LMFA) was proposed. LMFA combines a latent Markov model with mixture factor analysis: The Markov model captures changes in MMs over time by clustering subjects’ observations into a few states and state-specific factor analyses reveal what the MMs look like. However, to estimate the model, Vogelsmeier, Vermunt, van Roekel, and De Roover (2019) introduced a one-step (full information maximum likelihood; FIML) approach that is counterintuitive for applied researchers and entails cumbersome model selection procedures in the presence of many covariates. In this paper, we simplify the complex LMFA estimation and facilitate the exploration of covariate effects on state memberships by splitting the estimation in three intuitive steps: (1) obtain states with mixture factor analysis while treating repeated measures as independent, (2) assign observations to the states, and (3) use these states in a discrete- or continuous-time latent Markov model taking into account classification errors. A real data example demonstrates the empirical value.

Mixture factor analysis with distance metric constraint for dimensionality reduction

Dimensionality reduction (DR) is a key preprocessing stage in high-dimensional data classification. Traditional linear DR algorithms, e.g., Linear Discriminant Analysis, transform the original data into a low-dimensional subspace with a linear transformation matrix. However, these methods cannot handle complex nonlinearly separable data. Although some nonlinear DR methods, e.g., Locally Linear Embedding, are proposed to solve this problem, most of them are unsupervised, which only focus on the data structure hidden in the original high-dimensional space, rather than maximizing the inter-class separability of the transformed data, thus reducing the classification accuracy. To tackle this challenge, a novel supervised nonlinear DR algorithm, distance metric restricted mixture factor analysis (DMR-MFA), is proposed for high-dimensional data classification. In DMR-MFA, the original data is divided into several clusters, and the generation of original data in each cluster is described via a factor analysis model. Meanwhile, the distance metric constraint (DMC) is used for maximizing the separability of transformed low-dimensional data from different classes. Moreover, the optimal model parameters are learned via the joint optimization of log-likelihood function and DMC loss function, which makes the DMR-MFA possible to obtain the more separable low-dimensional embeddings while accurately describing the original data. Experimental results on synthetic data, benchmark datasets and high-resolution range profile data demonstrate that our method can handle nonlinearly separable data and improves the classification accuracy of data with high dimensionality.

Semantic Fisher Scores for Task Transfer: Using Objects to Classify Scenes.

The transfer of a neural network (CNN) trained to recognize objects to the task of scene classification is considered. A Bag-of-Semantics (BoS) representation is first induced, by feeding scene image patches to the object CNN, and representing the scene image by the ensuing bag of posterior class probability vectors (semantic posteriors). The encoding of the BoS with a Fisher vector (FV) is then studied. A link is established between the FV of any probabilistic model and the Q-function of the expectation-maximization (EM) algorithm used to estimate its parameters by maximum likelihood. This enables 1) immediate derivation of FVs for any model for which an EM algorithm exists, and 2) leveraging efficient implementations from the EM literature for the computation of FVs. It is then shown that standard FVs, such as those derived from Gaussian or even Dirichlet mixtures, are unsuccessful for the transfer of semantic posteriors, due to the highly non-linear nature of the probability simplex. The analysis of these FVs shows that significant benefits can ensue by 1) designing FVs in the natural parameter space of the multinomial distribution, and 2) adopting sophisticated probabilistic models of semantic feature covariance. The combination of these two insights leads to the encoding of the BoS in the natural parameter space of the multinomial, using a vector of Fisher scores derived from a mixture of factor analyzers (MFA). A network implementation of the MFA Fisher Score (MFA-FS), denoted as the MFAFSNet, is finally proposed to enable end-to-end training. Experiments with various object CNNs and datasets show that the approach has state-of-the-art transfer performance. Somewhat surprisingly, the scene classification results are superior to those of a CNN explicitly trained for scene classification, using a large scene dataset (Places). This suggests that holistic analysis is insufficient for scene classification. The modeling of local object semantics appears to be at least equally important. The two approaches are also shown to be strongly complementary, leading to very large scene classification gains when combined, and outperforming all previous scene classification approaches by a sizable margin.

Mixtures of factor analyzers with scale mixtures of fundamental skew normal distributions

Mixtures of factor analyzers (MFA) provide a powerful tool for modelling high-dimensional datasets. In recent years, several generalizations of MFA have been developed where the normality assumption of the factors and/or of the errors were relaxed to allow for skewness in the data. However, due to the form of the adopted component densities, the distribution of the factors/errors in most of these models is typically limited to modelling skewness concentrated in a single direction. Here, we introduce a more flexible finite mixture of factor analyzers based on the class of scale mixtures of canonical fundamental skew normal (SMCFUSN) distributions. This very general class of skew distributions can capture various types of skewness and asymmetry in the data. In particular, the proposed mixtures of SMCFUSN factor analyzers (SMCFUSNFA) can simultaneously accommodate multiple directions of skewness. As such, it encapsulates many commonly used models as special and/or limiting cases, such as models of some versions of skew normal and skew t-factor analyzers, and skew hyperbolic factor analyzers. For illustration, we focus on the t-distribution member of the class of SMCFUSN distributions, leading to mixtures of canonical fundamental skew t-factor analyzers (CFUSTFA). Parameter estimation can be carried out by maximum likelihood via an EM-type algorithm. The usefulness and potential of the proposed model are demonstrated using four real datasets.

Vector-Based Feature Representations for Speech Signals: From Supervector to Latent Vector

There are two basic types of feature representations for speech signals. The first type refers to probabilistic models, such as the Gaussian mixture model (GMM). The second type refers to vector-based feature representations, such as the Gaussian supervector (GSV). Since vector-based feature representations are easier to use and process, they are more popular than probabilistic model-based feature representations. In this paper, we begin by explaining the rationale behind two widely used vector-based feature representations, viz. GSV and the i-vector, and then make extensions. GSV is a supervector (SV) based on maximum a posteriori (MAP) adaptation. Its computation is simple and fast, but its dimensionality is high and fixed. While the i-vector is a latent vector (LV) based on factor analysis (FA). Although the computation can be time-consuming because of additional model parameters, its dimensionality is changeable. To generalize GSV, we propose the MAP SV, which is also based on MAP adaptation but can have an even higher dimensionality and thus carry more information. To boost the computational efficiency of the i-vector, we adopt the concept of the mixture of factor analyzers (MFA) and propose the MFA LV, which exhibits a similar flexibility in dimensionality but is faster in computation. The experimental results for speaker identification and verification tasks demonstrate that, MAP SV can be more robust than GSV, and MFALV is comparable to or even better than the i-vector in effectiveness and meanwhile maintains a higher computational efficiency. With a powerful backend, GSV and MAP SV are comparable to the i-vector and MFALV, but the latter two are more flexible in dimensionality.

Mixtures of factor analyzers with covariates for modeling multiply censored dependent variables

Censored data arise frequently in diverse applications in which observations to be measured may be subject to some upper and lower detection limits due to the restriction of experimental apparatus such that they are not exactly quantifiable. Mixtures of factor analyzers with censored data (MFAC) have been recently proposed for model-based density estimation and clustering of high-dimensional data in the presence of censored observations. In this paper, we consider an extended version of MFAC by considering regression equations to describe the relationship between covariates and multiply censored dependent variables. Two analytically feasible EM-type algorithms are developed for computing maximum likelihood estimates of model parameters with closed-form expressions. Moreover, we provide an information-based method to compute asymptotic standard errors of mixing proportions and regression coefficients. The utility and performance of our proposed methodology are illustrated through a simulation study and two real data examples.

Unsupervised and Supervised Feature Extraction Methods for Hyperspectral Images Based on Mixtures of Factor Analyzers

This paper proposes three feature extraction (FE) methods based on density estimation for hyperspectral images (HSIs). The methods are a mixture of factor analyzers (MFA), deep MFA (DMFA), and supervised MFA (SMFA). The MFA extends the Gaussian mixture model to allow a low-dimensionality representation of the Gaussians. DMFA is a deep version of MFA and consists of a two-layer MFA, i.e, samples from the posterior distribution at the first layer are input to an MFA model at the second layer. SMFA consists of single-layer MFA and exploits labeled information to extract features of HSI effectively. Based on these three FE methods, the paper also proposes a framework that automatically extracts the most important features for classification from an HSI. The overall accuracy of a classifier is used to automatically choose the optimal number of features and hence performs dimensionality reduction (DR) before HSI classification. The performance of MFA, DMFA, and SMFA FE methods are evaluated and compared to five different types of unsupervised and supervised FE methods by using four real HSIs datasets.

Approximating multivariate posterior distribution functions from Monte Carlo samples for sequential Bayesian inference.

An important feature of Bayesian statistics is the opportunity to do sequential inference: the posterior distribution obtained after seeing a dataset can be used as prior for a second inference. However, when Monte Carlo sampling methods are used for inference, we only have a set of samples from the posterior distribution. To do sequential inference, we then either have to evaluate the second posterior at only these locations and reweight the samples accordingly, or we can estimate a functional description of the posterior probability distribution from the samples and use that as prior for the second inference. Here, we investigated to what extent we can obtain an accurate joint posterior from two datasets if the inference is done sequentially rather than jointly, under the condition that each inference step is done using Monte Carlo sampling. To test this, we evaluated the accuracy of kernel density estimates, Gaussian mixtures, mixtures of factor analyzers, vine copulas and Gaussian processes in approximating posterior distributions, and then tested whether these approximations can be used in sequential inference. In low dimensionality, Gaussian processes are more accurate, whereas in higher dimensionality Gaussian mixtures, mixtures of factor analyzers or vine copulas perform better. In our test cases of sequential inference, using posterior approximations gives more accurate results than direct sample reweighting, but joint inference is still preferable over sequential inference whenever possible. Since the performance is case-specific, we provide an R package mvdens with a unified interface for the density approximation methods.

Automated learning of mixtures of factor analysis models with missing information

The mixture of factor analyzers (MFA) model has emerged as a useful tool to perform dimensionality reduction and model-based clustering for heterogeneous data. In seeking the most appropriate number of factors (q) of a MFA model with the number of components (g) fixed a priori, a two-stage procedure is commonly implemented by firstly carrying out parameter estimation over a set of prespecified numbers of factors, and then selecting the best q according to certain penalized likelihood criteria. When the dimensionality of data grows higher, such a procedure can be computationally prohibitive. To overcome this obstacle, we develop an automated learning scheme, called the automated MFA (AMFA) algorithm, to effectively merge parameter estimation and selection of q into a one-stage algorithm. The proposed AMFA procedure that allows for much lower computational cost is also extended to accommodate missing values. Moreover, we explicitly derive the score vector and the empirical information matrix for calculating standard errors associated with the estimated parameters. The potential and applicability of the proposed method are demonstrated through a number of real datasets with genuine and synthetic missing values.

Model-based clustering of censored data via mixtures of factor analyzers

Mixtures of factor analyzers (MFA) provide a promising tool for modeling and clustering high-dimensional data that contain an overwhelmingly large number of attributes measured on individuals arisen from a heterogeneous population. Due to the restriction of experimental apparatus, measurements can be limited to some lower and/or upper detection bounds and thus the data are possibly censored. In this paper, we extend the MFA to accommodate censored data, and the new model is called the MFA with censoring (MFAC). A computationally feasible alternating expectation conditional maximization (AECM) algorithm is developed to carry out maximum likelihood estimation of the MFAC model. Practical issues related to model-based clustering and recovery of censored data are also discussed. Simulation studies are conducted to examine the effect of censoring in classification, estimation and cluster validation. We also present an application of the proposed approach to two real data examples in which a certain number of left-censored observations are present.

Scalable learning and probabilistic analytics of industrial big data based on parameter server: Framework, methods and applications

With the ever increasing scale of industrial data, the computational burden for process modeling and analytics has becoming tremendous, particularly for large-scale processes. In this paper, a distributed parallel probabilistic learning framework based on scalable Parameter Server (PS) architecture is proposed for big process data. Under this framework, the traditional Variational Inference (VI) probabilistic model can be transformed to a scalable form through the Stochastic Variational Inference (SVI) algorithm, which can be further deployed on the PS architecture to make a distributed and parallel model. As an example, the traditional Variational Inference Mixture Factor Analysis (VIMFA) model is converted to the SVI-MFA model and deployed on the PS architecture for process big data modeling. Then it is utilized for process monitoring and quality prediction applications. A numerical case is first generated to validate the feasibility and efficiency of the SVI-MFA modeling algorithm, and then a TE benchmark process and a real Methanation Unit process demonstrate the effectiveness of the proposed framework for big process data modeling and analytics.

Mixtures of multivariate restricted skew-normal factor analyzer models in a Bayesian framework

The mixture of factor analyzers (MFA) model, by reducing the number of free parameters through its factor-analytic representation of the component covariance matrices, is an important statistical model to identify hidden or latent groups in high dimensional data. Recent approaches to extend the approach to skewed data or skewness in the latent groups have been examined in a frequentist setting where there are some known computational limitations. For these reasons we consider a Bayesian approach to the restricted skew-normal mixtures of factor analysis MFA model. We examine the performance and flexibility of the approach on real datasets and illustrate some of the computational advantages in a missing data setting.

Mixtures of Gaussian copula factor analyzers for clustering high dimensional data

Mixtures of factor analyzers is a useful model-based clustering method which can avoid the curse of dimensionality in high-dimensional clustering. However, this approach is sensitive to both diverse non-normalities of marginal variables and outliers, which are commonly observed in multivariate experiments. We propose mixtures of Gaussian copula factor analyzers (MGCFA) for clustering high-dimensional clustering. This model has two advantages; (1) it allows different marginal distributions to facilitate fitting flexibility of the mixture model, (2) it can avoid the curse of dimensionality by embedding the factor-analytic structure in the component-correlation matrices of the mixture distribution.An EM algorithm is developed for the fitting of MGCFA. The proposed method is free of the curse of dimensionality and allows any parametric marginal distribution which fits best to the data. It is applied to both synthetic data and a microarray gene expression data for clustering and shows its better performance over several existing methods.

Log Likelihood Monitoring for Multimode Process Using Variational Bayesian Mixture Factor Analysis Model

When a traditional mixture factor analysis (MFA) model is used for multimode process monitoring, the determination of parameter is complex, and the construction of monitoring statistics only considers the expectation in probability distributions of factor space and residual space. In this paper, a novel fault detection method based on a variational Bayesian MFA model for multimode process is introduced. The parameters of the MFA model structure, namely the number of local factor analyzer and the reduced dimensionality inside each factor analyzer, can be easily obtained through the birth-and-death Markov chain Monte Carlo algorithm and the variational inference technique. After parameter estimation for the Bayesian MFA model is done, a new monitoring index called negative variational log likelihood is developed by utilizing the whole information in probability distribution functions of all parameters. At last, two case studies, including a numerical example and the Tennessee Eastman (TE) process, verify the effectiveness and feasibility of the proposed monitoring scheme.