Mixture Of Factor Analyzers Research Articles

Flexible High-Dimensional Unsupervised Learning with Missing Data.

The mixture of factor analyzers (MFA) model is a famous mixture model-based approach for unsupervised learning with high-dimensional data. It can be useful, inter alia, in situations where the data dimensionality far exceeds the number of observations. In recent years, the MFA model has been extended to non-Gaussian mixtures to account for clusters with heavier tail weight and/or asymmetry. The generalized hyperbolic factor analyzers (MGHFA) model is one such extension, which leads to a flexible modelling paradigm that accounts for both heavier tail weight and cluster asymmetry. In many practical applications, the occurrence of missing values often complicates data analyses. A generalization of the MGHFA is presented to accommodate missing values. Under a missing-at-random mechanism, we develop a computationally efficient alternating expectation conditional maximization algorithm for parameter estimation of the MGHFA model with different patterns of missing values. The imputation of missing values under an incomplete-data structure of MGHFA is also investigated. The performance of our proposed methodology is illustrated through the analysis of simulated and real data.

A Novel Deep Density Model for Unsupervised Learning

Density models are fundamental in machine learning and have received a widespread application in practical cognitive modeling tasks and learning problems. In this work, we introduce a novel deep density model, referred to as deep mixtures of factor analyzers with common loadings (DMCFA), with an efficient greedy layer-wise unsupervised learning algorithm. The model employs a mixture of factor analyzers sharing common component loadings in each layer. The common loadings can be considered to be a feature selection or reduction matrix which makes this new model more physically meaningful. Importantly, sharing common components is capable of reducing both the number of free parameters and computation complexity remarkably. Consequently, DMCFA makes inference and learning rely on a dramatically more succinct model and avoids sacrificing its flexibility in estimating the data density by utilizing Gaussian distributions as the priors. Our model is evaluated on five real datasets and compared to three other competitive models including mixtures of factor analyzers (MFA), MFA with common loadings (MCFA), deep mixtures of factor analyzers (DMFA), and their collapsed counterparts. The results demonstrate the superiority of the proposed model in the tasks of density estimation, clustering, and generation.

What explains student’s intentions to pursue a certified professional accountancy qualification?

PurposeThis study examines the factors that explain Ghanaian business students’ intentions to pursue a Certified Professional Accounting (CPA) qualification.Design/methodology/approachThe study uses survey method of research and mainly used questionnaires as data collection instrument to elicit information from the respondent group. Using a total of 641 business students from the University of Ghana Business School, a mixture of factor analysis and logistic regression analysis techniques was used in investigating the study objectives.FindingsResults from the regression analysis demonstrate that students’ preference and beliefs about accounting professional qualification, academic major and grade point average are significantly associated with their intentions to pursue a CPA qualification. Moreover, majority of the sampled respondents (47.9 per cent) with the desire to pursue a CPA qualification prefer to be affiliated with the Association of Chartered Certified Accountants, 27.2 per cent prefer the Institute of Charted Accountant Ghana qualification, while only 7 per cent of the students prefer to be affiliated with the Chartered Institute of Management Accountant.Research limitations/implicationsThe findings from this study have important implications for the various professional accountancy bodies, accounting educators and other stakeholders interested in promoting the choice of accounting as a career path among university students.Practical implicationsThe results indicate that, as part of the broader strategy to increase the number of professional accountants in Ghana, there is the need to promote the study of accounting at the undergraduate level in the various universities in the country.Originality/valueThis study uniquely uses the behavioral decision theory to explain the dominant factors associated with students’ intentions to pursue a CPA qualification.

비대칭 혼합모형과 요인분석자 혼합모형을 이용한 VaR 추정

투자위험을 관리해야하는 금융기관에서 다수의 투자수익으로 구성된 포트폴리오 VaR (Value-at-Risk)를 정확히 추정하는 것은 매우 중요하다. VaR 예측을 위한 투자수익의 분포는 종종 두꺼운 꼬리를 갖고 있거나 왜도나 첨도가 존재한다. 본 연구에서는 총투자수익률의 VaR를 추정하기 위하여 MSN (mixtures of skew-normal distribution), MSt (mixtures of skew-t distribution) 등 비대칭혼합모형 뿐만 아니라, 고차원 자료 분포의 모수추정에 보다 절약적인 요인분석자 모형인 MFA (mixtures of factor analyzers), MCFA (mixtures of common factor analyzers), MCtFA (mixtures of common t factor analyzers) 등이 고려되었다. 한국 코스피 시장의 회사주식 수익률 자료를 이용하여 실증 분석한 결과 고려된 혼합모형들이 VaR 추정에 있어서 모두 비슷한 성능을 보여주고 있으나, MFA 모형과 MCtFA 모형이 다른 모형들에 비해서 상대적으로 자료의 경험적 VaR에 조금 더 가까운 추정값을 도출하였다.

A new two-layer mixture of factor analyzers with joint factor loading model for the classification of small dataset problems

Abstract Dimensionality Reduction (DR) is a fundamental topic of pattern classification and machine learning. For classification tasks, DR is typically employed as a pre-processing step, succeeded by an independent classifier training stage. However, such independent operation of the two stages often limits the final classification performance notably, as the generated subspace may not be maximally beneficial or appropriate to the learning task at hand. This problem is further accentuated for high-dimensional data classification in situations of the limited number of samples. To address this problem, we develop a novel joint learning model for classification, referred to as two-layer mixture of factor analyzers with joint factor loading (2L-MJFA). Specifically, the model adopts a special two-layer mixture or a mixture of mixtures structure, where each component represents each specific class as a mixture of factor analyzers (MFA). Importantly, all the involved factor analyzers are intentionally designed so that they share the same loading matrix. This, apart from operating as the DR matrix, largely reduces the parameters and makes the proposed algorithm very suitable to small dataset situations. Additionally, we propose a modified expectation maximization algorithm to train the proposed model. A series of simulation experiments demonstrate that what we propose significantly outperforms other state-of-the-art algorithms on various benchmark datasets. Finally, since factor analyzers are closely linked with Auto-encoder networks, the proposed idea could be of particular utility to the community of neural networks.

Robust, fuzzy, and parsimonious clustering, based on mixtures of Factor Analyzers

A clustering algorithm that combines the advantages of fuzzy clustering and robust statistical estimators is presented. It is based on mixtures of Factor Analyzers, endowed by the joint usage of impartial trimming and constrained estimation of scatter matrices, in a modified maximum likelihood approach. The algorithm generates a set of membership values, that are used to fuzzy partition the data set and to contribute to the robust estimates of the mixture parameters. The adoption of clusters modeled by Gaussian Factor Analysis allows for dimension reduction and for discovering local linear structures in the data. The new methodology has been shown to be resistant to different types of contamination, by applying it on artificial data. A brief discussion on the tuning parameters, such as the trimming level, the fuzzifier parameter, the number of clusters and the value of the scatter matrices constraint, has been developed, also with the help of some heuristic tools for their choice. Finally, a real data set has been analyzed, to show how intermediate membership values are estimated for observations lying at cluster overlap, while cluster cores are composed by observations that are assigned to a cluster in a crisp way.

Probability density estimation in sensor networks based on distributed mixture of factor analyzers, mobile agents and stochastic sensor selection

This paper considers the problem of distributed probability density estimation of high-dimensional data in sensor networks. In order to describe and analyze high-dimensional observations, a mixture of factor analyzers can be used instead of Gaussian mixture model. Due to high communication costs between sensor nodes in centralized algorithms, use of these algorithms is not affordable. In this paper, a distributed estimation algorithm is presented based on the mixture of factor analyzers, mobile agents and stochastic sensor selection. In the proposed algorithm, at the beginning of each iteration, a mobile agent is assigned to each independent route of the network which consists of several sensor nodes based on a stochastic sensor selection scheme. The mobile agents calculate local sufficient statistics vector in each sensor node and update global sufficient statistics. At the end of each iteration, the parameters of the mixture model are computed by using global sufficient statistics. Convergence analysis of the proposed distributed algorithm is also presented. Finally, the performance of the proposed algorithm is evaluated by using numerical simulations. Simulation results show the promising performance of the proposed distributed algorithm.

Left/Right or U? Estimating the Dimensionality of National Party Competition in Europe

Comparative studies report the rise of left- and right-wing Eurosceptic parties that have transformed national party competition in Europe toward an inverted U-shaped configuration: peripheral parties at the left and right of the party spectrum oppose while centrist parties support several features of European integration. To describe the tempo and timing of this transformation and the heterogeneity across countries, we construct a Bayesian finite mixture factor analysis that estimates the election-specific probability of a one-dimensional left/right versus a two-dimensional inverted U-shaped national party configuration. The results show a general trend toward “U” but with significant variation across countries and time, including cases with a reversal of this trend.

A silent speech system based on permanent magnet articulography and direct synthesis

In this paper we present a silent speech interface (SSI) system aimed at restoring speech communication for individuals who have lost their voice due to laryngectomy or diseases affecting the vocal folds. In the proposed system, articulatory data captured from the lips and tongue using permanent magnet articulography (PMA) are converted into audible speech using a speaker-dependent transformation learned from simultaneous recordings of PMA and audio signals acquired before laryngectomy. The transformation is represented using a mixture of factor analysers, which is a generative model that allows us to efficiently model non-linear behaviour and perform dimensionality reduction at the same time. The learned transformation is then deployed during normal usage of the SSI to restore the acoustic speech signal associated with the captured PMA data. The proposed system is evaluated using objective quality measures and listening tests on two databases containing PMA and audio recordings for normal speakers. Results show that it is possible to reconstruct speech from articulator movements captured by an unobtrusive technique without an intermediate recognition step. The SSI is capable of producing speech of sufficient intelligibility and naturalness that the speaker is clearly identifiable, but problems remain in scaling up the process to function consistently for phonetically rich vocabularies.

Semiadaptive Fault Diagnosis via Variational Bayesian Mixture Factor Analysis with Application to Wastewater Treatment

Mainly due to the hostile environment in wastewater plants (WWTPs), the reliability of sensors with respect to important qualities is often poor. In this work, we present the design of a semiadaptive fault diagnosis method based on the variational Bayesian mixture factor analysis (VBMFA) to support process monitoring. The proposed method is capable of capturing strong nonlinearity and the significant dynamic feature of WWTPs that seriously limit the application of conventional multivariate statistical methods for fault diagnosis implementation. The performance of proposed method is validated through a simulation study of a wastewater plant. Results have demonstrated that the proposed strategy can significantly improve the ability of fault diagnosis under fault-free scenario, accurately detect the abrupt change and drift fault, and even localize the root cause of corresponding fault properly.

Extending mixtures of factor models using the restricted multivariate skew-normal distribution

The mixture of factor analyzers (MFA) model provides a powerful tool for analyzing high-dimensional data as it can reduce the number of free parameters through its factor-analytic representation of the component covariance matrices. This paper extends the MFA model to incorporate a restricted version of the multivariate skew-normal distribution to model the distribution of the latent component factors, called mixtures of skew-normal factor analyzers (MSNFA). The proposed MSNFA model allows us to relax the need for the normality assumption for the latent factors in order to accommodate skewness in the observed data. The MSNFA model thus provides an approach to model-based density estimation and clustering of high-dimensional data exhibiting asymmetric characteristics. A computationally feasible ECM algorithm is developed for computing the maximum likelihood estimates of the parameters. Model selection can be made on the basis of three commonly used information-based criteria. The potential of the proposed methodology is exemplified through applications to two real examples, and the results are compared with those obtained from fitting the MFA model.

Distributed Density Estimation Based on a Mixture of Factor Analyzers in a Sensor Network.

Distributed density estimation in sensor networks has received much attention due to its broad applicability. When encountering high-dimensional observations, a mixture of factor analyzers (MFA) is taken to replace mixture of Gaussians for describing the distributions of observations. In this paper, we study distributed density estimation based on a mixture of factor analyzers. Existing estimation algorithms of the MFA are for the centralized case, which are not suitable for distributed processing in sensor networks. We present distributed density estimation algorithms for the MFA and its extension, the mixture of Student’s t-factor analyzers (MtFA). We first define an objective function as the linear combination of local log-likelihoods. Then, we give the derivation process of the distributed estimation algorithms for the MFA and MtFA in details, respectively. In these algorithms, the local sufficient statistics (LSS) are calculated at first and diffused. Then, each node performs a linear combination of the received LSS from nodes in its neighborhood to obtain the combined sufficient statistics (CSS). Parameters of the MFA and the MtFA can be obtained by using the CSS. Finally, we evaluate the performance of these algorithms by numerical simulations and application example. Experimental results validate the promising performance of the proposed algorithms.

Research on dimension reduction method for hyperspectral remote sensing image based on global mixture coordination factor analysis

Abstract. Over the past thirty years, the hyperspectral remote sensing technology is attracted more and more attentions by the researchers. The dimension reduction technology for hyperspectral remote sensing image data is one of the hotspots in current research of hyperspectral remote sensing. In order to solve the problems of nonlinearity, the high dimensions and the redundancy of the bands that exist in the hyperspectral data, this paper proposes a dimension reduction method for hyperspectral remote sensing image data based on the global mixture coordination factor analysis. In the first place, a linear low dimensional manifold is obtained from the nonlinear and high dimensional hyperspectral image data by mixture factor analysis method. In the second place, the parameters of linear low dimensional manifold are estimated by the EM algorithm of find a local maximum of the data log-likelihood. In the third place, the manifold is aligned to a global parameterization by the global coordinated factor analysis model and then the lowdimension image data of hyperspectral image data is obtained at last. Through the comparison of different dimensionality reduction method and different classification method for the low-dimensional data, the result illuminates the proposed method can retain maximum spectral information in hyperspectral image data and can eliminate the redundant among bands.

Further advances on Bayesian Ying-Yang harmony learning

After a short tutorial on the fundamentals of Bayes approaches and Bayesian Ying-Yang (BYY) harmony learning, this paper introduces new progresses. A generic information harmonising dynamics of BYY harmony learning is proposed with the help of a Lagrange variety preservation principle, which provides Lagrange-like implementations of Ying-Yang alternative nonlocal search for various learning tasks and unifies attention, detection, problem-solving, adaptation, learning and model selection from an information harmonising perspective. In this framework, new algorithms are developed to implement Ying-Yang alternative nonlocal search for learning Gaussian mixture and several typical exemplars of linear matrix system, including factor analysis (FA), mixture of local FA, binary FA, nonGaussian FA, de-noised Gaussian mixture, sparse multivariate regression, temporal FA and temporal binary FA, as well as a generalised bilinear matrix system that covers not only these linear models but also manifold learning, gene regulatory networks and the generalised linear mixed model. These algorithms are featured with a favourable nature of automatic model selection and a unified formulation in performing unsupervised learning and semi-supervised learning. Also, we propose a principle of preserving multiple convex combinations, which leads alternative search algorithms. Finally, we provide a chronological outline of the history of BYY learning studies.

Learning local factor analysis versus mixture of factor analyzers with automatic model selection

Considering Factor Analysis (FA) for each component of Gaussian Mixture Model (GMM), clustering and local dimensionality reduction can be addressed simultaneously by Mixture of Factor Analyzers (MFA) and Local Factor Analysis (LFA), which correspond to two FA parameterizations, respectively. This paper investigates the performance of Variational Bayes (VB) and Bayesian Ying-Yang (BYY) harmony learning on MFA/LFA for the problem of automatically determining the component number and the local hidden dimensionalities (i.e., the number of factors of FA in each component). Similar to the existing VB learning algorithm on MFA, we develop an alternative VB algorithm on LFA with a similar conjugate Dirichlet–Normal–Gamma (DNG) prior on all parameters of LFA. Also, the corresponding BYY algorithms are developed for MFA and LFA. A wide range of synthetic experiments shows that LFA is superior to MFA in model selection under either VB or BYY, while BYY outperforms VB reliably on both MFA and LFA. These empirical findings are consistently observed from real applications on not only face and handwritten digit images clustering, but also unsupervised image segmentation.

Multimode Process Monitoring Based on Aligned Mixture Factor Analysis

To meet the demand for diversified products, industrial processes typically have multiple operating modes. In this article, a novel scheme based on aligned mixture factor analysis (AMFA) is proposed for multimode process monitoring. First, mixture factor analysis (MFA) is applied to produce a statistical fingerprint of the training data that gives a detailed description of the multiple operating modes. Then, unlike conventional multimodal algorithms in which multiple models are constructed and monitoring results are softly combined, the proposed method aims at aligning the separated local models together and performing the monitoring behaviors based on the global model. Through this approach of dividing and integrating, both the within-mode and cross-mode correlations can be greatly preserved in the global model. Finally, the utility and feasibility of the proposed method are demonstrated through a numerical example, a nonisothermal continuous stirred-tank reactor (CSTR) model, and the Tennessee Eastman benchmark process.

A machine learning approach using EEG data to predict response to SSRI treatment for major depressive disorder

ObjectiveThe problem of identifying, in advance, the most effective treatment agent for various psychiatric conditions remains an elusive goal. To address this challenge, we investigate the performance of the proposed machine learning (ML) methodology (based on the pre-treatment electroencephalogram (EEG)) for prediction of response to treatment with a selective serotonin reuptake inhibitor (SSRI) medication in subjects suffering from major depressive disorder (MDD). MethodsA relatively small number of most discriminating features are selected from a large group of candidate features extracted from the subject’s pre-treatment EEG, using a machine learning procedure for feature selection. The selected features are fed into a classifier, which was realized as a mixture of factor analysis (MFA) model, whose output is the predicted response in the form of a likelihood value. This likelihood indicates the extent to which the subject belongs to the responder vs. non-responder classes. The overall method was evaluated using a “leave-n-out” randomized permutation cross-validation procedure. ResultsA list of discriminating EEG biomarkers (features) was found. The specificity of the proposed method is 80.9% while sensitivity is 94.9%, for an overall prediction accuracy of 87.9%. There is a 98.76% confidence that the estimated prediction rate is within the interval [75%, 100%]. ConclusionsThese results indicate that the proposed ML method holds considerable promise in predicting the efficacy of SSRI antidepressant therapy for MDD, based on a simple and cost-effective pre-treatment EEG. SignificanceThe proposed approach offers the potential to improve the treatment of major depression and to reduce health care costs.

Subspace K-means clustering

To achieve an insightful clustering of multivariate data, we propose subspace K-means. Its central idea is to model the centroids and cluster residuals in reduced spaces, which allows for dealing with a wide range of cluster types and yields rich interpretations of the clusters. We review the existing related clustering methods, including deterministic, stochastic, and unsupervised learning approaches. To evaluate subspace K-means, we performed a comparative simulation study, in which we manipulated the overlap of subspaces, the between-cluster variance, and the error variance. The study shows that the subspace K-means algorithm is sensitive to local minima but that the problem can be reasonably dealt with by using partitions of various cluster procedures as a starting point for the algorithm. Subspace K-means performs very well in recovering the true clustering across all conditions considered and appears to be superior to its competitor methods: K-means, reduced K-means, factorial K-means, mixtures of factor analyzers (MFA), and MCLUST. The best competitor method, MFA, showed a performance similar to that of subspace K-means in easy conditions but deteriorated in more difficult ones. Using data from a study on parental behavior, we show that subspace K-means analysis provides a rich insight into the cluster characteristics, in terms of both the relative positions of the clusters (via the centroids) and the shape of the clusters (via the within-cluster residuals).

CHull as an alternative to AIC and BIC in the context of mixtures of factor analyzers

Mixture analysis is commonly used for clustering objects on the basis of multivariate data. When the data contain a large number of variables, regular mixture analysis may become problematic, because a large number of parameters need to be estimated for each cluster. To tackle this problem, the mixtures-of-factor-analyzers (MFA) model was proposed, which combines clustering with exploratory factor analysis. MFA model selection is rather intricate, as both the number of clusters and the number of underlying factors have to be determined. To this end, the Akaike (AIC) and Bayesian (BIC) information criteria are often used. AIC and BIC try to identify a model that optimally balances model fit and model complexity. In this article, the CHull (Ceulemans & Kiers, 2006) method, which also balances model fit and complexity, is presented as an interesting alternative model selection strategy for MFA. In an extensive simulation study, the performances of AIC, BIC, and CHull were compared. AIC performs poorly and systematically selects overly complex models, whereas BIC performs slightly better than CHull when considering the best model only. However, when taking model selection uncertainty into account by looking at the first three models retained, CHull outperforms BIC. This especially holds in more complex, and thus more realistic, situations (e.g., more clusters, factors, noise in the data, and overlap among clusters).

Mixture Factor Analysis for Approximating a Nonnormally Distributed Continuous Latent Factor With Continuous and Dichotomous Observed Variables

Mixture factor analysis is examined as a means of flexibly estimating nonnormally distributed continuous latent factors in the presence of both continuous and dichotomous observed variables. A simulation study compares mixture factor analysis with normal maximum likelihood (ML) latent factor modeling. Different results emerge for continuous versus dichotomous outcomes. For dichotomous outcomes, normal ML path estimates have bias that worsens as latent factor skew/kurtosis increases and does not diminish as sample size increases, whereas the mixture factor analysis model produces nearly unbiased estimators as sample sizes increase (500 and greater) and offers near nominal coverage probability. For continuous outcome variables, both methods produce factor loading estimates with minimal bias regardless of latent factor skew, but the mixture factor analysis is more efficient. The method is demonstrated using data motivated by a study on youth with cystic fibrosis examining predictors of treatment adherence. In summary, mixture factor analysis provides improvements over normal ML estimation in the presence of skewed/kurtotic latent factors, but due to variability in the estimator relating the latent factor to dichotomous outcomes and computational issues, the improvements were only fully realized, in this study, at larger sample sizes (500 and greater).