Multivariate Normal Mixture Model Research Articles

Penalized maximum likelihood estimator for finite multivariate skew normal mixtures

In practice, multivariate skew normal mixture (MSNM) models provide a more flexible framework than multivariate normal mixture models, especially for heterogeneous and asymmetric data. For MSNM models, the maximum likelihood estimator often leads to a statistical inference referred to as “badness” under certain properties, because of the unboundedness of the likelihood function and the divergence of shape parameters. We consider two penalties for the log-likelihood function to counter these issues simultaneously in MSNM models. We show that the penalized maximum likelihood estimator is strongly consistent when the putative order of the mixture is equal to or larger than the true order. We also provide penalized expectation-maximization-type algorithms to compute penalized estimates. Finite sample performance is examined through simulations, real data applications, and comparison with existing methods.

A Likelihood Ratio Test of a Homoscedastic Multivariate Normal Mixture Against a Heteroscedastic Multivariate Normal Mixture

The multivariate finite normal mixture model is one of the most commonly used tools to analyze a heterogeneous data. When using the multivariate finite normal mixture model, one is usually interested in knowing whether a homoscedastic mixture model can be used to simplify the model. The likelihood ratio test (LRT) is the most popular statistic tool to choose between two nested models. Under the null model of a homoscedastic multivariate normal mixture, the asymptotic χ2 distribution is commonly used to approximate the null distribution of the LRT statistic. However, it is demonstrated using numerical studies that the χ2 distribution approximation is not satisfactory and fails to control the nominal type I error unless the sample size is larger than 2000, the mixture components are well-separated, and the singular solutions are avoided. A parametric bootstrap method is further proposed to approximate the distribution of the LRT statistic and its effectiveness is evaluated through extensive numerical studies.

A new class of multivariate goodness of fit tests for multivariate normal mixtures

The goodness of fit (GOF) test plays an important role to determine whether a data set comes from an assumed distribution (with possibly unknown parameters). Many approaches have been proposed to perform the goodness of fit test, such as Chi-square, Kolmogorov-Smirnov, Cramér-Von Mises, and Anderson Darling test statistics, but most of those are proposed for univariate data. In this paper, we propose a new class of goodness of fit tests for assessing multivariate normal mixtures (with multivariate normality as a special case) based on dimension reduction. The new method allows us to directly employ existing univariate GOF tests to perform multivariate GOF tests after linearly projecting multivariate data to univariate data. The new GOF statistic is defined as the maximum of some one-dimensional GOF test statistics. We propose a bootstrap procedure to approximate the distribution of the new GOF statistic. Our approach is simple, general and can be easily extended to perform GOF tests for many other multivariate distributions. The effectiveness of our proposed new GOF tests for multivariate normal mixture models is demonstrated via simulation studies and a real data application.

Incorporating customer personalization preferences in open product architecture design

Open product architecture is a key enabler for product personalization, as it allows the integration of personalized modules in a product architecture to satisfy individual customer needs and preference. A critical challenge for integrating personalized modules into a product architecture is determining the optimal assembly architecture when considering market expectations and manufacturing constraints. In this paper, an optimization method is proposed for determining the personalized product design architecture that incorporates individual customer preferences. First, a decision hierarchy is presented to describe the integrated design decisions of the product architecture, including product variety determination, module variant selection, and personalized module configuration. Next, a profit model is formulated as an overall performance metric that incorporates customer preferences and manufacturing cost. The systematic patterns and randomness of diverse customer preferences are modeled by combining conjoint analysis and market segmentation with a multivariate normal mixture model. Individual customer product utilities in the target market and their product purchase intent probability are estimated through Monte-Carlo simulation, which is incorporated into the profit calculation. Manufacturing limitations on processes and materials are included as they influence manufacturer’s planning on candidate module variants and production strategies of personalized modules. These models are used to determine a product family architecture that maximizes profit by optimally determining its offering of product variants, module combinations, and personalized module configuration through a genetic algorithm. The proposed method is demonstrated by a personalized bicycle architecture design example.

Analytical expression for the integrated squared density partial derivative of a multivariate normal mixture distribution

ABSTRACTThis paper describes the derivation of the analytical expression for the integrated squared density partial derivative (ISDPD) in a multivariate normal mixture model. The analytical expression of the ISDPD is derived for arbitrary dimensions with partial derivative orders up to four. Although the value of the ISDPD can be obtained by using the common numerical integration via mathematical software (such as Maple, Mathematica, Matlab, etc), it suffers from the limitation of computation time when the dimension or the number of mixture components of the considered multivariate normal mixture model are large. Moreover, numerical comparison indicates the benefits of speed offered by our proposed analytical expression are far superior to the numerical integration performed by Maple. With this analytical expression, the ISDPD can apace be calculated with no assistance of numerical integration.

A latent class analysis of PTSD symptoms among inner city primary care patients

ObjectivesExamine evidence for different subclasses of posttraumatic stress disorder (PTSD) symptoms in a sample of trauma exposed, low-income, predominantly African American men and women. Assess the relationship between PTSD subclasses with major depressive disorder (MDD) and types of trauma experienced. MethodLatent class analysis (LCA) using a multivariate normal mixture model on the 17-item PTSD Symptom Scale (PSS) was used to identify latent subclasses of PTSD symptoms (N = 5063). ResultsLCA suggested four subclasses of PTSD symptoms: (1) High severity and comorbidity (n = 932, 92.2% current PTSD, 88.7% MDD, 82% both), characterized by high PTSD symptoms, depression, and comorbidity of PTSD and MDD; (2) Moderate severity (n = 1179, 56.5% current PTSD, 53.9% MDD, 34.5% both), which had high avoidance and hyper-vigilance symptoms compared to the other symptoms; (3) Low PTSD and high depression (n = 657, 12.8% current PTSD, 49.9% MDD, 8.8% both) which had high insomnia but otherwise low PTSD symptoms and high depression; and (4) Resilient (n = 2295, 2.0% current PTSD, 16.4% MDD, and 0.6% both) characterized by low mean scores on all PTSD symptoms and depression. ConclusionsThe results suggest avoidance and hyper-vigilance are important symptoms in PTSD development and insomnia may be an important indicator for depression. The combination of severe insomnia, avoidance, and hyper-vigilance may be key symptoms for comorbidity of PTSD and MDD. Future studies should focus on these symptoms to better target people at high risk for developing PTSD or MDD.

Multivariate normal mean-variance mixture distribution based on Lindley distribution

ABSTRACTThis article introduces a new asymmetric distribution constructed by assuming the multivariate normal mean-variance mixture model. Called normal mean-variance mixture of the Lindley distribution, we derive some mathematical properties of the new distribution. Also, a feasible maximum likelihood estimation procedure using the EM algorithm and the asymptotic standard errors of parameter estimates are developed. The performance of the proposed distribution is illustrated by means of real datasets and simulation analysis.

Detecting discordance enrichment among a series of two-sample genome-wide expression data sets

BackgroundWith the current microarray and RNA-seq technologies, two-sample genome-wide expression data have been widely collected in biological and medical studies. The related differential expression analysis and gene set enrichment analysis have been frequently conducted. Integrative analysis can be conducted when multiple data sets are available. In practice, discordant molecular behaviors among a series of data sets can be of biological and clinical interest.MethodsIn this study, a statistical method is proposed for detecting discordance gene set enrichment. Our method is based on a two-level multivariate normal mixture model. It is statistically efficient with linearly increased parameter space when the number of data sets is increased. The model-based probability of discordance enrichment can be calculated for gene set detection.ResultsWe apply our method to a microarray expression data set collected from forty-five matched tumor/non-tumor pairs of tissues for studying pancreatic cancer. We divided the data set into a series of non-overlapping subsets according to the tumor/non-tumor paired expression ratio of gene PNLIP (pancreatic lipase, recently shown it association with pancreatic cancer). The log-ratio ranges from a negative value (e.g. more expressed in non-tumor tissue) to a positive value (e.g. more expressed in tumor tissue). Our purpose is to understand whether any gene sets are enriched in discordant behaviors among these subsets (when the log-ratio is increased from negative to positive). We focus on KEGG pathways. The detected pathways will be useful for our further understanding of the role of gene PNLIP in pancreatic cancer research. Among the top list of detected pathways, the neuroactive ligand receptor interaction and olfactory transduction pathways are the most significant two. Then, we consider gene TP53 that is well-known for its role as tumor suppressor in cancer research. The log-ratio also ranges from a negative value (e.g. more expressed in non-tumor tissue) to a positive value (e.g. more expressed in tumor tissue). We divided the microarray data set again according to the expression ratio of gene TP53. After the discordance enrichment analysis, we observed overall similar results and the above two pathways are still the most significant detections. More interestingly, only these two pathways have been identified for their association with pancreatic cancer in a pathway analysis of genome-wide association study (GWAS) data.ConclusionsThis study illustrates that some disease-related pathways can be enriched in discordant molecular behaviors when an important disease-related gene changes its expression. Our proposed statistical method is useful in the detection of these pathways. Furthermore, our method can also be applied to genome-wide expression data collected by the recent RNA-seq technology.

Exogenous impacts on the links between energy and agricultural commodity markets

The main purpose of this paper is to identify the effects of exogenous factors, which have been somewhat controversial, on the price links between the energy and agricultural commodity markets. Our study differs from other studies by employing multivariate normal mixture models to capture the structural properties of the price dependencies in the underlying states. This paper investigates price dependencies from both quantitative and structural perspectives. By analyzing the overall dependencies and structural heterogeneity in the empirical results, we conclude that the global financial crisis is the most influential shock on the price links between energy and agricultural commodities. Because price links are vulnerable to financial shocks, our results also suggest introducing state-based analysis to risk management and portfolio diversification across the energy and agriculture markets during times of turmoil.

Unrestricted Mixture Models for Class Identification in Growth Mixture Modeling

Growth mixture modeling has gained much attention in applied and methodological social science research recently, but the selection of the number of latent classes for such models remains a challenging issue, especially when the assumption of proper model specification is violated. The current simulation study compared the performance of a linear growth mixture model (GMM) for determining the correct number of latent classes against a completely unrestricted multivariate normal mixture model. Results revealed that model convergence is a serious problem that has been underestimated by previous GMM studies. Based on two ways of dealing with model nonconvergence, the performance of the two types of mixture models and a number of model fit indices in class identification are examined and discussed. This article provides suggestions to practitioners who want to use GMM for their research.

Concordant integrative gene set enrichment analysis of multiple large-scale two-sample expression data sets

BackgroundGene set enrichment analysis (GSEA) is an important approach to the analysis of coordinate expression changes at a pathway level. Although many statistical and computational methods have been proposed for GSEA, the issue of a concordant integrative GSEA of multiple expression data sets has not been well addressed. Among different related data sets collected for the same or similar study purposes, it is important to identify pathways or gene sets with concordant enrichment.MethodsWe categorize the underlying true states of differential expression into three representative categories: no change, positive change and negative change. Due to data noise, what we observe from experiments may not indicate the underlying truth. Although these categories are not observed in practice, they can be considered in a mixture model framework. Then, we define the mathematical concept of concordant gene set enrichment and calculate its related probability based on a three-component multivariate normal mixture model. The related false discovery rate can be calculated and used to rank different gene sets.ResultsWe used three published lung cancer microarray gene expression data sets to illustrate our proposed method. One analysis based on the first two data sets was conducted to compare our result with a previous published result based on a GSEA conducted separately for each individual data set. This comparison illustrates the advantage of our proposed concordant integrative gene set enrichment analysis. Then, with a relatively new and larger pathway collection, we used our method to conduct an integrative analysis of the first two data sets and also all three data sets. Both results showed that many gene sets could be identified with low false discovery rates. A consistency between both results was also observed. A further exploration based on the KEGG cancer pathway collection showed that a majority of these pathways could be identified by our proposed method.ConclusionsThis study illustrates that we can improve detection power and discovery consistency through a concordant integrative analysis of multiple large-scale two-sample gene expression data sets.

A Bernoulli–Gaussian mixture model of donation likelihood and monetary value: An application to alumni segmentation in a university setting

Advances in computational power and enterprise technology, e.g., Customer Relationship Management (CRM) software and data warehouses, allow many businesses to collect a wealth of information on large numbers of consumers. This includes information on past purchasing behavior, demographic characteristics, as well as how consumers interact with the organization, e.g., in events, on the web. The ability to mine such data sets is crucial to an organization’s ability to deliver better customer service, as well as manage its resource allocation decisions. To this end, we formulate a Bernoulli–Gaussian mixture model that jointly describes the likelihood and monetary value of repeat transactions. In addition to presenting the model, we derive an instance of the Expectation–Maximization Algorithm to estimate the associated parameters, and to segment the consumer population.We apply the model to an extensive dataset of donations received at a private, Ph.D.-granting university in the Midwestern United States. We use the model to assess the effect of individual traits on their contribution likelihood and monetary value, discuss insights stemming from the results, and how the model can be used to support resource allocation decisions. For example, we find that participation in alumni-oriented activities, i.e., reunions or travel programs, is associated with increased donation likelihood and value, and that fraternity/sorority membership magnifies this effect. The presence/characterization of unobserved, cross-sectional heterogeneity in the data set, i.e., unobserved/unexplained systematic differences among individuals, is, perhaps, our most important finding. Finally, we argue that the proposed segmentation approach is more appealing than alternatives appearing in the literature that consider donation likelihood and monetary value separately. Among them and as a benchmark, we compare the proposed model to a segmentation that builds on a multivariate Normal mixture model, and conclude that the Bernoulli–Gaussian mixture model provides a more coherent approach to generate segments.

Modeling contribution behavior in fundraising: Segmentation analysis for a public broadcasting station

Funding pressures have forced many not-for-profit organizations to reduce their reliance on mass-marketing efforts, e.g., pledge drives, and increase the volume and sophistication of their direct marketing activities. The efficiency of direct marketing, however, is linked to an organization’s ability to target population segments effectively, which, in turn, has motivated the development of methodological approaches for market segmentation.In this paper, we use finite-mixture models as a framework to analyze member contributions at a public radio station in the Midwestern United States between 2001 and 2008. We exploit the methodology’s flexibility, in a novel setting, to formulate segmentation models under different assumptions about the processes that generate the individual contribution sequences, i.e., longitudinal data. This differs from conventional models summarizing transactions with aggregate statistics, i.e., RFM data, that can introduce bias. We first consider generalized (Multivariate) Normal mixture models to obtain segmentations based on contribution frequencies and monetary value. To capture serial dependence in the contribution sequences, we present Markov Chain mixture models yielding segmentations based on behavioral patterns. We also derive instances of the Expectation–Maximization algorithm to estimate the associated parameters. Rather than modeling how segment size and membership evolve, we assume stable membership, and attribute manifest changes to the characteristics that define the segments. This, supports the identification of systematic, but unobserved, differences between individuals, and enables optimization of marketing policies for the ensuing segments.Analysis of the data from the public radio station shows significant heterogeneity in excess of that explained by residence location – a proxy for socioeconomic traits. In terms of managerial insights, we conclude that the contributions are largely driven by recommended membership levels, which suggests that the radio station can exert influence to increase revenue. We also find that the public radio members exhibit three distinct behavioral patterns: a “Low Variance” segment associated with individuals who consistently contribute at a given level, a “Transient” segment describing the behavior of individuals who allow their membership to remain lapsed, and a “High Variance” segment for individuals whose contribution frequencies and levels exhibit significant variation, which according to the station’s management, could be a manifestation of propensity to donate in response to solicitations.

On-line EM Variants for Multivariate Normal Mixture Model in Background Learning and Moving Foreground Detection

The unsupervised learning of multivariate mixture models from on-line data streams has attracted the attention of researchers for its usefulness in real-time intelligent learning systems. The EM algorithm is an ideal choice for iteratively obtaining maximum likelihood estimation of parameters in presumable finite mixtures, comparing to some popular numerical methods. However, the original EM is a batch algorithm that works only on fixed datasets. To endow the EM algorithm with the capability to process streaming data, two on-line variants are studied, including Titterington's method and a sufficient statistics-based method. We first prove that the two on-line EM variants are theoretically feasible for training the multivariate normal mixture model by showing that the model belongs to the exponential family. Afterward, the two on-line learning schemes for multivariate normal mixtures are applied to the problems of background learning and moving foreground detection. Experiments show that the two on-line EM variants can efficiently update the parameters of the mixture model and are capable of generating reliable backgrounds for moving foreground detection.

Reject Inference of Incomplete Data Using a Normal Mixture Model

Reject inference in credit scoring is a statistical approach to adjust for nonrandom sample bias due to rejected applicants. Function estimation approaches are based on the assumption that rejected applicants are not necessary to be included in the estimation, when the missing data mechanism is missing at random. On the other hand, the density estimation approach by using mixture models indicates that reject inference should include rejected applicants in the model. When mixture models are chosen for reject inference, it is often assumed that data follow a normal distribution. If data include missing values, an application of the normal mixture model to fully observed cases may cause another sample bias due to missing values. We extend reject inference by a multivariate normal mixture model to handle incomplete characteristic variables. A simulation study shows that inclusion of incomplete characteristic variables outperforms the function estimation approaches.

Statistical models for temporal variations of seismicity parameters to forecast seismicity rates in Japan

This paper introduces a model to forecast the rate of earthquakes for a specified period and area. The model explicitly predicts the number of earthquakes and b-value of the Gutenberg-Richter distribution for the period of interest with an autoregressive process. The model also incorporates a time dependency adjustment for higher magnitude ranges, assuming that as time passes since the last large earthquake within the area, the probability of another larger earthquake increases. These predictions are overlaid on a spatial density map obtained with a multivariate normal mixture model of the historical earthquakes that have occurred in the area. This forecast model differs from currently proposed models by its density estimation and its assumption of temporal changes. The model has been submitted to the Earthquake Forecasting Testing Experiment for Japan.

Maximum Likelihood Estimation of the Multivariate Normal Mixture Model

The Hessian of the multivariate normal mixture model is derived, and estimators of the information matrix are obtained, thus enabling consistent estimation of all parameters and their precisions. The usefulness of the new theory is illustrated with two examples and some simulation experiments. The newly proposed estimators appear to be superior to the existing ones.

Real-time anomaly detection in hyperspectral images using multivariate normal mixture models and GPU processing

Hyperspectral imaging, which records a detailed spectrum of light arriving in each pixel, has many potential uses in remote sensing as well as other application areas. Practical applications will typically require real-time processing of large data volumes recorded by a hyperspectral imager. This paper investigates the use of graphics processing units (GPU) for such real-time processing. In particular, the paper studies a hyperspectral anomaly detection algorithm based on normal mixture modelling of the background spectral distribution, a computationally demanding task relevant to military target detection and numerous other applications. The algorithm parts are analysed with respect to complexity and potential for parallellization. The computationally dominating parts are implemented on an Nvidia GeForce 8800 GPU using the Compute Unified Device Architecture programming interface. GPU computing performance is compared to a multi-core central processing unit implementation. Overall, the GPU implementation runs significantly faster, particularly for highly data-parallelizable and arithmetically intensive algorithm parts. For the parts related to covariance computation, the speed gain is less pronounced, probably due to a smaller ratio of arithmetic to memory access. Detection results on an actual data set demonstrate that the total speedup provided by the GPU is sufficient to enable real-time anomaly detection with normal mixture models even for an airborne hyperspectral imager with high spatial and spectral resolution.

The full Bayesian significance test for mixture models: results in gene expression clustering

Gene clustering is a useful exploratory technique to group together genes with similar expression levels under distinct cell cycle phases or distinct conditions. It helps the biologist to identify potentially meaningful relationships between genes. In this study, we propose a clustering method based on multivariate normal mixture models, where the number of clusters is predicted via sequential hypothesis tests: at each step, the method considers a mixture model of m components (m = 2 in the first step) and tests if in fact it should be m - 1. If the hypothesis is rejected, m is increased and a new test is carried out. The method continues (increasing m) until the hypothesis is accepted. The theoretical core of the method is the full Bayesian significance test, an intuitive Bayesian approach, which needs no model complexity penalization nor positive probabilities for sharp hypotheses. Numerical experiments were based on a cDNA microarray dataset consisting of expression levels of 205 genes belonging to four functional categories, for 10 distinct strains of Saccharomyces cerevisiae. To analyze the method's sensitivity to data dimension, we performed principal components analysis on the original dataset and predicted the number of classes using 2 to 10 principal components. Compared to Mclust (model-based clustering), our method shows more consistent results.

Cluster analysis of tooth size in subjects with normal occlusion

Tooth-size studies are important in orthodontics. Our aim in this study was to establish normative data on tooth size with a clustering method. Dental casts of 307 subjects with normal occlusion were examined. In these subjects, the tooth-size data sets in the maxilla and the mandible were clustered for men and women by using multivariate normal mixture models. This analysis clustered the data sets into groups of observations that were cohesive and separate from the other groups. The parameters were estimated with expectation-maximization procedures, and the number of clusters was selected via the Bayesian information criterion. Seven heterogeneous clusters in men and 4 clusters in women were identified. This study indicated that, in normal occlusion, tooth sizes are variable. These heterogeneities in the normal occlusion group seem to have contributed to the variability found in normative data. The method used in this study seems to provide a more substantive design for artificial teeth and add an additional dimension in the process of diagnosis of patients. Further applications seem possible in dental anthropometry by simultaneously dealing with the full dentition as a data set.