Mixture Of Multivariate Normal Distributions Research Articles

A Comparison of Information Criteria in Clustering Based on Mixture of Multivariate Normal Distributions

Clustering analysis based on a mixture of multivariate normal distributions is commonly used in the clustering of multidimensional data sets. Model selection is one of the most important problems in mixture cluster analysis based on the mixture of multivariate normal distributions. Model selection involves the determination of the number of components (clusters) and the selection of an appropriate covariance structure in the mixture cluster analysis. In this study, the efficiency of information criteria that are commonly used in model selection is examined. The effectiveness of information criteria has been determined according to the success in the selection of the number of components and in the selection of an appropriate covariance matrix.

Bayes shrinkage estimation for high-dimensional VAR models with scale mixture of normal distributions for noise

We propose Bayesian shrinkage methods for coefficient estimation for high-dimensional vector autoregressive (VAR) models using scale mixtures of multivariate normal distributions for independently sampled additive noises. We also suggest an efficient selection procedure for the shrinkage parameter as a computationally feasible alternative to the traditional MCMC sampling methods for high-dimensional data. A shrinkage parameter is selected at the minimum point of a newly proposed score function which is asymptotically equivalent to the mean squared error of the model coefficients. The selected shrinkage parameter is presented in a closed form as a function of sample size, level of noise, and non-normality in data, and it can be efficiently estimated by using a suggested variation of cross validation. Consistency of both of the cross validation estimator and proposed shrinkage estimator is proved. The competitiveness of the proposed methods is demonstrated based on comprehensive experimental results using simulated data and high-dimensional plant gene expression data in the context of coefficient estimation and structural inference for VAR models. The proposed methods are applicable to high-dimensional stationary time series with or without near unit roots.

Simulation of non-stationary wind speed and direction time series

The simulation of synthetic wind time series is strictly necessary, among others, to assess the risk of structures and systems, to model the dynamic of morphological features like dunes, or to investigate the potential of wind energy in a certain location. For all these applications, a tool able to simulate time series of wind speed and direction is required. In this paper we propose a methodology for the simulation of bivariate non-stationary time series of wind speed and direction. This methodology takes into account the circular nature of the wind direction, as well as the mean annual cycle of both wind speed and direction. For modelling the joint distribution of the two variables, wind speed is modelled conditioned to the wind direction. For modelling short-term self- and cross-dependency among the variables a Vector Autoregressive (VAR) model of order p is used, with an innovation process at time t that depends on the value taken by the variables at time t−1, based on the use of a mixture of multivariate normal distributions. The proposed methodology is applied to a case study, simulating several synthetic series, of the same length as the original series. The simulated series satisfactorily captures the characteristics of averaging or common data (i.e. with low or moderate speed) and, under certain independence conditions, the directional extreme value distribution of the wind speed is also properly reproduced.

Efficient Sampling Methods for Truncated Multivariate Normal and Student-tDistributions Subject to Linear Inequality Constraints

Sampling from a truncated multivariate distribution subject to multiple linear inequality constraints is a recurring problem in many areas in statistics and econometrics, such as the order-restricted regressions, censored data models, and shape-restricted nonparametric regressions. However, the sampling problem remains nontrivial due to the analytically intractable normalizing constant of the truncated multivariate distribution. We first develop an efficient rejection sampling method for the truncated univariate normal distribution, and analytically establish its superiority in terms of acceptance rates compared to some of the popular existing methods. We then extend our methodology to obtain samples from a truncated multivariate normal distribution subject to convex polytope restriction regions. Finally, we generalize the sampling method to truncated scale mixtures of multivariate normal distributions. Empirical results are presented to illustrate the superior performance of our proposed Gibbs sampler in...

Clustering with the multivariate normal inverse Gaussian distribution

Many model-based clustering methods are based on a finite Gaussian mixture model. The Gaussian mixture model implies that the data scatter within each group is elliptically shaped. Hence non-elliptical groups are often modeled by more than one component, resulting in model over-fitting. An alternative is to use a mean–variance mixture of multivariate normal distributions with an inverse Gaussian mixing distribution (MNIG) in place of the Gaussian distribution, to yield a more flexible family of distributions. Under this model the component distributions may be skewed and have fatter tails than the Gaussian distribution. The MNIG based approach is extended to include a broad range of eigendecomposed covariance structures. Furthermore, MNIG models where the other distributional parameters are constrained is considered. The Bayesian Information Criterion is used to identify the optimal model and number of mixture components. The method is demonstrated on three sample data sets and a novel variation on the univariate Kolmogorov–Smirnov test is used to assess goodness of fit.

Multiple Imputation of Missing or Faulty Values Under Linear Constraints

Many statistical agencies, survey organizations, and research centers collect data that suffer from item nonresponse and erroneous or inconsistent values. These data may be required to satisfy linear constraints, for example, bounds on individual variables and inequalities for ratios or sums of variables. Often these constraints are designed to identify faulty values, which then are blanked and imputed. The data also may exhibit complex distributional features, including nonlinear relationships and highly nonnormal distributions. We present a fully Bayesian, joint model for modeling or imputing data with missing/blanked values under linear constraints that (i) automatically incorporates the constraints in inferences and imputations, and (ii) uses a flexible Dirichlet process mixture of multivariate normal distributions to reflect complex distributional features. Our strategy for estimation is to augment the observed data with draws from a hypothetical population in which the constraints are not present, thereby taking advantage of computationally expedient methods for fitting mixture models. Missing/blanked items are sampled from their posterior distribution using the Hit-and-Run sampler, which guarantees that all imputations satisfy the constraints. We illustrate the approach using manufacturing data from Colombia, examining the potential to preserve joint distributions and a regression from the plant productivity literature. Supplementary materials for this article are available online.

Measures of multivariate skewness and kurtosis in high-dimensional framework

Skewness and kurtosis characteristics of a multivariate p-dimensional distribution introduced by Mardia (1970) have been used in various testing procedures and demonstrated attractive asymptotic properties in large sample settings. However these characteristics are not designed for high-dimensional problems where the dimensionality, p can largely exceeds the sample size, N. Such type of high-dimensional data are commonly encountered in modern statistical applications. This the suggests that new measures of skewness and kurtosis that can accommodate high-dimensional settings must be derived and carefully studied. In this paper, we show that, by exploiting the dependence structure, new expressions for skewness and kurtosis are introduced as an extension of the corresponding Mardia’s measures, which uses the potential advantages that the block-diagonal covariance structure has to offer in high dimensions. Asymptotic properties of newly derived measures are investigated and the cumulant based characterizations are presented along with of applications to a mixture of multivariate normal distributions and multivariate Laplace distribution, for which the explicit expressions of skewness and kurtosis are obtained. Test statistics based on the new measures of skewness and kurtosis are proposed for testing a distribution shape, and their limit distributions are established in the asymptotic framework where N→∞ and p is fixed but large, including p>N. For the dependence structure learning, the gLasso based technique is explored followed by AIC step which we propose for optimization of the gLasso candidate model. Performance accuracy of the test procedures based on our estimators of skewness and kurtosis are evaluated using Monte Carlo simulations and the validity of the suggested approach is shown for a number of cases when p>N.

A multivariate spatial mixture model for areal data: examining regional differences in standardized test scores.

Researchers in the health and social sciences often wish to examine joint spatial patterns for two or more related outcomes. Examples include infant birth weight and gestational length, psychosocial and behavioral indices, and educational test scores from different cognitive domains. We propose a multivariate spatial mixture model for the joint analysis of continuous individual-level outcomes that are referenced to areal units. The responses are modeled as a finite mixture of multivariate normals, which accommodates a wide range of marginal response distributions and allows investigators to examine covariate effects within subpopulations of interest. The model has a hierarchical structure built at the individual level (i.e., individuals are nested within areal units), and thus incorporates both individual- and areal-level predictors as well as spatial random effects for each mixture component. Conditional autoregressive (CAR) priors on the random effects provide spatial smoothing and allow the shape of the multivariate distribution to vary flexibly across geographic regions. We adopt a Bayesian modeling approach and develop an efficient Markov chain Monte Carlo model fitting algorithm that relies primarily on closed-form full conditionals. We use the model to explore geographic patterns in end-of-grade math and reading test scores among school-age children in North Carolina.

MRHMMs: Multivariate Regression Hidden Markov Models and the variantS

Hidden Markov models (HMMs) are flexible and widely used in scientific studies. Particularly in genomics and genetics, there are multiple distinct regimes in the genome within each of which the relationships among multivariate features are distinct. Examples include differential gene regulation depending on gene functions and experimental conditions, and varying combinatorial patterns of multiple transcription factors. We developed a software package called MRHMMs (Multivariate Regression Hidden Markov Models and the variantS) that accommodates a variety of HMMs that can be flexibly applied to many biological studies and beyond. MRHMMs supplements existing HMM software packages in two aspects. First, MRHMMs provides a diverse set of emission probability structures, including mixture of multivariate normal distributions and (logistic) regression models. Second, MRHMMs is computationally efficient for analyzing large data-sets generated in current genome-wide studies. Especially, the software is written in C for the speed advantage and further amenable to implement alternative models to meet users' own purposes. http://sourceforge.net/projects/mrhmms/

Bayesian analysis for nonlinear mixed‐effects models under heavy‐tailed distributions

A common assumption in nonlinear mixed-effects models is the normality of both random effects and within-subject errors. However, such assumptions make inferences vulnerable to the presence of outliers. More flexible distributions are therefore necessary for modeling both sources of variability in this class of models. In the present paper, I consider an extension of the nonlinear mixed-effects models in which random effects and within-subject errors are assumed to be distributed according to a rich class of parametric models that are often used for robust inference. The class of distributions I consider is the scale mixture of multivariate normal distributions that consist of a wide range of symmetric and continuous distributions. This class includes heavy-tailed multivariate distributions, such as the Student's t and slash and contaminated normal. With the scale mixture of multivariate normal distributions, robustification is achieved from the tail behavior of the different distributions. A Bayesian framework is adopted, and MCMC is used to carry out posterior analysis. Model comparison using different criteria was considered. The procedures are illustrated using a real dataset from a pharmacokinetic study. I contrast results from the normal and robust models and show how the implementation can be used to detect outliers.

On a Screened Pearson Correlation and Its Application to Interdependence Methods

This article derives and examines the screened correlation obtained by applying a screening scheme to a scale mixture of multivariate normal distributions. We investigate and quantify the effects of the screening scheme on correlations and consider some properties and inferential aspects of the screened correlation while developing necessary theories and providing illustrative examples. The results indicate that the screening scheme is a useful technique for multivariate interdependence methods, particularly for addressing the problem of a lack of correlation.

On the asymptotic behaviour of the pseudolikelihood ratio test statistic with boundary problems

This paper considers the asymptotic distribution of the likelihood ratio statistic T for testing a subset of parameter of interest θ, θ = (γ, η), H(0) : γ = γ(0), based on the pseudolikelihood L(θ, ϕ̂), where ϕ̂ is a consistent estimator of ϕ, the nuisance parameter. We show that the asymptotic distribution of T under H(0) is a weighted sum of independent chi-squared variables. Some sufficient conditions are provided for the limiting distribution to be a chi-squared variable. When the true value of the parameter of interest, θ(0), or the true value of the nuisance parameter, ϕ(0), lies on the boundary of parameter space, the problem is shown to be asymptotically equivalent to the problem of testing the restricted mean of a multivariate normal distribution based on one observation from a multivariate normal distribution with misspecified covariance matrix, or from a mixture of multivariate normal distributions. A variety of examples are provided for which the limiting distributions of T may be mixtures of chi-squared variables. We conducted simulation studies to examine the performance of the likelihood ratio test statistics in variance component models and teratological experiments.

Geometrical Recombination Operators for Real-Coded Evolutionary MCMCs

Markov chain Monte Carlo (MCMC) algorithms are sampling methods for intractable distributions. In this paper, we propose and investigate algorithms that improve the sampling process from multi-dimensional real-coded spaces. We present MCMC algorithms that run a population of samples and apply recombination operators in order to exchange useful information and preserve commonalities in highly probable individual states. We call this class of algorithms Evolutionary MCMCs (EMCMCs). We introduce and analyze various recombination operators which generate new samples by use of linear transformations, for instance, by translation or rotation. These recombination methods discover specific structures in the search space and adapt the population samples to the proposal distribution. We investigate how to integrate recombination in the MCMC framework to sample from a desired distribution. The recombination operators generate individuals with a computational effort that scales linearly in the number of dimensions and the number of parents. We present results from experiments conducted on a mixture of multivariate normal distributions. These results show that the recombinative EMCMCs outperform the standard MCMCs for target distributions that have a nontrivial structural relationship between the dimensions.

Model-based discriminant analysis of Iberian potential vegetation and bio-climatic indices

While vegetation is seldom at equilibrium, equilibrium models of the vegetation-climate relationships point towards states that can be considered as asymptotic or potential. Provided that the newer conditions are within the range of those for which the models have been conceived and parameterised, equilibrium models can be used to estimate the impact of different climate scenarios, as they provide the newer trends towards newer equilibrium points. In this contribution, we present our on-going research on a statistical model applied to the Iberian vegetation and present initial results. We use a climatic data-set of monthly fields of precipitation and mean temperature and a vegetation map of Spain. We transform the climatic data into bio-climatic indexes and run a discriminant analysis for a subset of training sites. We discuss the ordination, parameterise as a mixture of multi-variate Gaussian distributions and generate a new bio-climatic map. While still have some more tuning to do, we are close to be ready to apply the same statistical model to newer climate scenarios.

Mixtures of regression models for time course gene expression data: evaluation of initialization and random effects

Finite mixture models are routinely applied to time course microarray data. Due to the complexity and size of this type of data, the choice of good starting values plays an important role. So far initialization strategies have only been investigated for data from a mixture of multivariate normal distributions. In this work several initialization procedures are evaluated for mixtures of regression models with and without random effects in an extensive simulation study on different artificial datasets. Finally, these procedures are also applied to a real dataset from Escherichia coli. The latest release versions of R packages flexmix, gcExplorer and kernlab are always available from CRAN (http://cran.r-project.org/). Supplementary data are available at Bioinformatics online.

Multiple-subjects connectivity-based parcellation using hierarchical Dirichlet process mixture models

We propose a hierarchical infinite mixture model approach to address two issues in connectivity-based parcellations: (i) choosing the number of clusters, and (ii) combining data from different subjects. In a Bayesian setting, we model voxel-wise anatomical connectivity profiles as an infinite mixture of multivariate Gaussian distributions, with a Dirichlet process prior on the cluster parameters. This type of prior allows us to conveniently model the number of clusters and estimate its posterior distribution directly from the data. An important benefit of using Bayesian modelling is the extension to multiple subjects clustering via a hierarchical mixture of Dirichlet processes. Data from different subjects are used to infer on class parameters and the number of classes at individual and group level. Such a method accounts for inter-subject variability, while still benefiting from combining different subjects data to yield more robust estimates of the individual clusterings.

PET Image Denoising Using a Synergistic Multiresolution Analysis of Structural (MRI/CT) and Functional Datasets

PET allows the imaging of functional properties of the living tissue, whereas other modalities (CT, MRI) provide structural information at significantly higher resolution and better image quality. Constraints for injected radioactivity, technologic limitations of current instrumentation, and inherent spatial uncertainties on the decaying process affect the quality of PET images. In this article we illustrate how structural information of matched anatomic images can be used in a multiresolution model to enhance the signal-to-noise ratio of PET images. The model states a flexible relation between function and structure in the brain and replaces high-resolution information of PET images with appropriately scaled MRI or CT local detail. The method can be naturally extended to other functional imaging modalities (SPECT, functional MRI). The methodology is based on the multiresolution property of the wavelet transform (WT). First, the coregistered structural image (MRI/CT) is downgraded to the resolution of the PET volume through appropriate filtering. Second, a redundant version of the WT is applied to both volumes. Third, a linear model is applied to the set of local coefficients of both image volumes and resulting parameters are recorded. The overall set of linear coefficients is then modeled as a mixture of multivariate gaussian distributions and fitted through a k-means algorithm. Finally, the local wavelet coefficients of the PET image are substituted by the corresponding values of the MRI/CT set calibrated according to the resulting clustering. The methodology was validated on digital simulated images and clinical data to evaluate its quantitative potential for individual as well as group analysis. Application to real and simulated datasets shows very effective noise reduction (15% SD) while resolution is preserved. The methodology is robust to errors in the coregistration parameters, practical to implement, and computationally fast.

Comparison of Supervised Clustering Methods for the Analysis of DNA Microarray Expression Data

Several typical supervised clustering methods such as Gaussian mixture model-based supervised clustering (GMM), k-nearest-neighbor (KNN), binary support vector machines (SVMs) and multiclass support vector machines (MC-SVMs) were employed to classify the computer simulation data and two real microarray expression datasets. False positive, false negative, true positive, true negative, clustering accuracy and Matthews' correlation coefficient (MCC) were compared among these methods. The results are as follows: (1) In classifying thousands of gene expression data, the performances of two GMM methods have the maximal clustering accuracy and the least overall FP+FN error numbers on the basis of the assumption that the whole set of microarray data are a finite mixture of multivariate Gaussian distributions. Furthermore, when the number of training sample is very small, the clustering accuracy of GMM-II method has superiority over GMM-I method. (2) In general, the superior classification performance of the MC-SVMs are more robust and more practical, which are less sensitive to the curse of dimensionality, and not only next to GMM method in clustering accuracy to thousands of gene expression data, but also more robust to a small number of high-dimensional gene expression samples than other techniques. (3) Of the MC-SVMs, OVO and DAGSVM perform better on the large sample sizes, whereas five MC-SVMs methods have very similar performance on moderate sample sizes. In other cases, OVR, WW and CS yield better results when sample sizes are small. So, it is recommended that at least two candidate methods, choosing on the basis of the real data features and experimental conditions, should be performed and compared to obtain better clustering result.

Model-Based Clustering With Dissimilarities: A Bayesian Approach

A Bayesian model-based clustering method is proposed for clustering objects on the basis of dissimilarites. This combines two basic ideas. The first is that the objects have latent positions in a Euclidean space, and that the observed dissimilarities are measurements of the Euclidean distances with error. The second idea is that the latent positions are generated from a mixture of multivariate normal distributions, each one corresponding to a cluster. We estimate the resulting model in a Bayesian way using Markov chain Monte Carlo. The method carries out multidimensional scaling and model-based clustering simultaneously, and yields good object configurations and good clustering results with reasonable measures of clustering uncertainties. In the examples we study, the clustering results based on low-dimensional configurations were almost as good as those based on high-dimensional ones. Thus, the method can be used as a tool for dimension reduction when clustering high-dimensional objects, which may be useful especially for visual inspection of clusters.We also propose a Bayesian criterion for choosing the dimension of the object configuration and the number of clusters simultaneously. This is easy to compute and works reasonably well in simulations and real examples.

On symmetry of finite mixtures of normal distributions

In this note we derive a necessary and sufficient condition for a distribution obtained by taking a finite mixture of multivariate normal distributions to be symmetric about zero. The result derived also holds for mixtures of symmetric stable distributions, including the Cauchy distribution.