Mixture Of Multivariate Normal Distributions Research Articles

Parsimonious mixture‐of‐experts based on mean mixture of multivariate normal distributions

The mixture‐of‐experts (MoE) paradigm attempts to learn complex models by combining several “experts” via probabilistic mixture models. Each expert in the MoE model handles a small area of the data space in which a gating function controls the data‐to‐expert assignment. The MoE framework has been used extensively in designing non‐linear models in machine learning and statistics to model the heterogeneity in data for the purpose of regression, classification and clustering. The existing MoE of multi‐target regression (MoE‐MTR) models for continuous data is based on multivariate normal distributions. However, in many practical situations, for a set of data, a group or groups of observations may exhibit asymmetric and heavy‐tailed behaviour, and inference based on symmetric distributions in such situations can unduly affect the fit of the regression model. We introduce here a novel robust multivariate non‐normal MoE model by the use of mean mixture of normal distributions. The proposed model can handle the issues of MoE‐MTR models regarding possibly skewed, heavy‐tailed and noisy data. Maximum likelihood estimates of model parameters are developed based on an expectation‐maximization (EM)‐type algorithm. Parsimony is also obtained by imposing suitable constraints on the expert dispersion matrices. The usefulness of the proposed methodology is illustrated using simulated and real data sets.

Family of extended mean mixtures of multivariate normal distributions: Properties, inference and applications

<abstract><p>A new class of skewed distributions, with a matrix skewness parameter, called extended mean mixtures of multivariate normal (EMMN) distributions, is constructed. The family of EMMN distributions includes the SN and MMN distributions as special cases. Some basic properties of this family, such as characteristic function, moment generating function, affine transformation and canonical forms of the distributions are derived. An EM-type algorithm is developed to carry out the maximum likelihood estimation of the parameters. Two special cases of this family are studied in detail. A simulation is carried out to examine the performance of the estimation method, and the flexibility is illustrated by fitting a special case of this family to a real data. Finally, the theoretical formula of the multivariate tail conditional expectation of the EMMN distribution is derived.</p></abstract>

Abstract 9855: Bayesian Multivariate Genetic Analysis of Lipids Improves Translational Insight

Introduction: Genome-wide association studies (GWAS) of blood lipids have yielded several translational insights for coronary artery disease. Hypothesis: We tested the hypothesis that novel Bayesian methods leveraging genetic correlation can improve insights from lipid genetics analyses. Methods: We used published lipids GWAS results from 297,626 participants of the Million Veterans Program (MVP), and generated summary statistics for 276,096 participants of the UK Biobank. We employed a multivariate adaptive shrinkage (mashR) model to each dataset. MashR allows the effect of a SNP to be modeled as a mixture of multivariate normal distributions across lipid levels (LDL-C, HDL-C, Total Cholesterol, and triglycerides) and an empirical Bayes approach to appropriately nudge the posterior estimates of the effect in accordance with the overall patterns observed in the data. Results: In addition to reproducing all 318 loci significantly associated with lipids in MVP, we identified 4,689 novel loci associated with lipids using mashR. We observed a high degree of agreement between MVP and UK Biobank, with over 95% of UKBB replicated in MVP and an additional 1246 identified. In genetic prioritization analyses, mashR shows improvement in the detection of known Mendelian genes and the consistency of ranks among subgroups compared with univariate results. Our approach improves the estimate of enrichment parameters of known genomic features similar to established GWAS variants. Finally, combining these joint estimates with tools for incorporating the genetic correlation structure, mashR improves the proportion of variation explained by lipid polygenic risk scores by up to 58% (se 0.015) when compared to univariate substrates across lipid traits. This improvement holds across both European and non-European individuals. We show that much of this gain in power arises from improved precision of the posterior estimates where the ratio of original standard error over posterior marginal variance reflects an increased ‘effective sample size’. Conclusions: Our framework shows that Bayesian multivariate genetic analysis markedly improves power for genomic discovery, better identifies causal genes, and improves polygenic risk prediction for complex traits.

A Comparative Study of Imputation Methods for Multivariate Ordinal Data

Abstract Missing data remains a very common problem in large datasets, including survey and census data containing many ordinal responses, such as political polls and opinion surveys. Multiple imputation (MI) is usually the go-to approach for analyzing such incomplete datasets, and there are indeed several implementations of MI, including methods using generalized linear models, tree-based models, and Bayesian non-parametric models. However, there is limited research on the statistical performance of these methods for multivariate ordinal data. In this article, we perform an empirical evaluation of several MI methods, including MI by chained equations (MICE) using multinomial logistic regression models, MICE using proportional odds logistic regression models, MICE using classification and regression trees, MICE using random forest, MI using Dirichlet process (DP) mixtures of products of multinomial distributions, and MI using DP mixtures of multivariate normal distributions. We evaluate the methods using simulation studies based on ordinal variables selected from the 2018 American Community Survey. Under our simulation settings, the results suggest that MI using proportional odds logistic regression models, classification and regression trees, and DP mixtures of multinomial distributions generally outperform the other methods. In certain settings, MI using multinomial logistic regression models is able to achieve comparable performance, depending on the missing data mechanism and amount of missing data.

A formulation for continuous mixtures of multivariate normal distributions

Several formulations have long existed in the literature in the form of continuous mixtures of normal variables where a mixing variable operates on the mean or on the variance or on both the mean and the variance of a multivariate normal variable, by changing the nature of these basic constituents from constants to random quantities. More recently, other mixture-type constructions have been introduced, where the core random component, on which the mixing operation operates, is not necessarily normal. The main aim of the present work is to show that many existing constructions can be encompassed by a formulation where normal variables are mixed using two univariate random variables. For this formulation, we derive various general properties, with focus on the multivariate context. Within the proposed framework, it is also simpler to formulate new proposals of parametric families, and we provide a few such instances. As a side product, the exposition provides a concise compendium of the main constructions of continuous normal-mixtures type, although a full overview of this vast theme is not attempted.

Estimation and Influence Diagnostics for the Multivariate Linear Regression Models with Skew Scale Mixtures of Normal Distributions

In this paper, we present recent results in the context of multivariate linear regression models considering that random errors follow multivariate skew scale mixtures of normal distributions. This class of distributions includes the scale mixtures of multivariate normal distributions, as special cases, and provides flexibility in capturing a wide variety of asymmetric behaviors. We implemented the algorithm ECM (Expectation/Conditional Maximization) and we obtained closed-form expressions for all the estimators of the parameters of the proposed model. Inspired by the ECM algorithm, we have developed an influence diagnostics for detecting influential observations to investigate the sensitivity of the maximum likelihood estimators. To examine the performance and the usefulness of the proposed methodology, we present simulation studies and analyze a real dataset.

Gaussian Mixture Model Based Soil Classification Using Multiple Cone Penetration Tests

This study presents an application of the Gaussian mixture (GM) method for soil classification using multiple cone penetration tests (CPT). Compared to the hard clustering methods, the GM model classifies the CPT data by representing the probability density function of observed variables as a mixture of multivariate normal distributions. A GM model based expectation maximization (EM) algorithm with Bayesian information criterion (BIC) for selecting the optimal number of clusters is developed using six real CPT data performed at Dunkerque site in the north of France. The classification results are compared with the classical CPT based interpretation using the non normalized soil behavior type (SBT) index together with the Robertson chart. The results show that the GM model is able to identify accurately the soil layers. In addition, the combination of all CPTs, rather than considering them separately, may improve the soil layers identification because all the site information is considered.

Non-deterministic and emotional chatting machine: learning emotional conversation generation using conditional variational autoencoders

Conversational responses are non-trivial for artificial conversational agents. Artificial responses should not only be meaningful and plausible, but should also (1) have an emotional context and (2) should be non-deterministic (i.e., vary given the same input). The two factors enumerated, respectively, above are involved and this is demonstrated such that previous studies have tackled them individually. This paper is the first to tackle them together. Specifically, we present two models both based upon conditional variational autoencoders. The first model learns disentangled latent representations to generate conversational responses given a specific emotion. The other model explicitly learns different emotions using a mixture of multivariate Gaussian distributions. Experiments show that our proposed models can generate more plausible and diverse conversation responses in accordance with designated emotions compared to baseline approaches.

Patterns recognition methods to study genotypic similarity in flood-irrigated rice

ABSTRACT Genetic diversity studies are performed based on information on a set of traits measured in a group of genotypes, considering one or more environments. The pattern recognition methods allow classifying genotypes from a set of important agronomic information. Thus, this study aimed to present and compare pattern recognition methods to inquire about the similarity of environments and genotypes in flood-irrigated rice for the recommendation of cultivars. The experim>ents were performed in the municipalities of Leopoldina, Lambari, and Janauba, state of Minas Gerais, Brazil. To evaluate the pattern of similarity, 25 rice genotypes in three environments belonging to the flood-irrigated rice breeding program were used. Among these genotypes, five cultivars were used as an experimental control for the grain yield, the height of the plant, flowering, panicle length, grains filled by panicles, percentages of grains filled by panicles, in the 2012/2013 agricultural year. The methods us>ed were mixtures of multivariate normal distributions and density-based clustering algorithm. It was observed, therefore, that the genotypes are distributed in three distinct groups, in which there are intragroup homogeneity and intergroup heterogeneity for the agronomic traits of the flooded rice culture. The methods >used to assess the dissimilarity of environments using pattern recognition methods were efficient in classifying flooded rice irrigated environments.

MICROSIMULATION MODEL CALIBRATION USING INCREMENTAL MIXTURE APPROXIMATE BAYESIAN COMPUTATION.

Microsimulation models (MSMs) are used to inform policy by predicting population-level outcomes under different scenarios. MSMs simulate individual-level event histories that mark the disease process (such as the development of cancer) and the effect of policy actions (such as screening) on these events. MSMs often have many unknown parameters; calibration is the process of searching the parameter space to select parameters that result in accurate MSM prediction of a wide range of targets. We develop Incremental Mixture Approximate Bayesian Computation (IMABC) for MSM calibration, which results in a simulated sample from the posterior distribution of model parameters given calibration targets. IMABC begins with a rejection-based ABC step, drawing a sample of points from the prior distribution of model parameters and accepting points that result in simulated targets that are near observed targets. Next, the sample is iteratively updated by drawing additional points from a mixture of multivariate normal distributions and accepting points that result in accurate predictions. Posterior estimates are obtained by weighting the final set of accepted points to account for the adaptive sampling scheme. We demonstrate IMABC by calibrating CRC-SPIN 2.0, an updated version of a MSM for colorectal cancer (CRC) that has been used to inform national CRC screening guidelines.

Novel Study of Model-Based Clustering Time Series Gene Expression in Different Tissues: Applications to Aging Process.

Aging is an organized biological process that is regulated by highly interconnected pathways between different cells and tissues in the living organism. Identification of similar genes between tissues in different ages may also help to discover the general mechanism of aging or to discover more effective therapeutic decisions. According to the wide application of model-based clustering techniques, the aim is to evaluate the performance of the Mixture of Multivariate Normal Distributions (MMNDs) as a valid method for clustering time series gene expression data with the Mixture of Matrix-Variate Normal Distributions (MMVNDs). In this study, the expression of aging data from NCBI's Gene Expression Omnibus was elaborated to utilize proper data. A set of common genes which were differentially expressed between different tissues were selected and then clustered together through two methods. Finally, the biological significance of clusters was evaluated, using their ability to find genes in the cell using Enricher. The MMVNDs is more efficient to find co-express genes. Six clusters of genes were observed using the MMVNDs. According to the functional analysis, most genes in clusters 1-6 are related to the B-cell receptors and IgG immunoglobulin complex, proliferating cell nuclear antigen complex, the metabolic pathways of iron, fat, and body mass control, the defense against bacteria, the cancer development incidence, and the chronic kidney failure, respectively. Results showed that most biological changes of aging between tissues are related to the specific components of immune cells. Also, the application of MMVNDs can increase the ability to find similar genes.

Model selection in biological networks using a graphical EM algorithm

The analysis of biological networks is an important task in life sciences. Most of biological interactions can be modeled using graphical networks where arcs represent probabilistic relationships between nodes or variables. Such models help scientists to analyze their complex data sets, test candidate interaction networks and understand the studied relationships. These studies face two major problems: the selection of most probable interaction topologies and the clustering of the associated peculiar data. In this paper, we model biological interactions with a mixture of multivariate Gaussian distributions. We, then, introduce a new algorithm for the parameters estimation and data clustering. This algorithm, called Graphical Expectation Maximization (GEM), extends the EM algorithm by taking into account several decomposable graph structures and using an original initialization technique. Applying this algorithm, we propose a model selection procedure based on the Bayesian Information Criterion. The accuracy of the proposed method is demonstrated on the grounds of a simulation study of a signal transduction network of the epidermal growth factor (EGFR) protein. Moreover, we apply the proposed model selection procedure to choose the most appropriate interaction graphs for microbial community in infant gut using a real data set.

Sample selection in the face of design constraints: Use of clustering to define sample strata for qualitative research.

To sample 40 physician organizations stratified on the basis of longitudinal cost of care measures for qualitative interviews in order to describe the range of care delivery structures and processes that are being deployed to influence the total costs of caring for patients. Three years of physician organization-level total cost of care data (n=156 in California) from the Integrated Healthcare Association's value-based pay-for-performance program. We fit total cost of care data using mixture and K-means clustering algorithms to segment the population of physician organizations into sampling strata based on 3-year cost trajectories (ie, cost curves). A mixture of multivariate normal distributions can classify physician organization cost curves into clusters defined by total cost level, shape, and within-cluster variation. K-means clustering does not accommodate differing levels of within-cluster variation and resulted in more clusters being allocated to unstable cost curves. A mixture of regressions approach focuses overly on anomalous trajectories and is sensitive to model coding. Statistical clustering can be used to form sampling strata when longitudinal measures are of primary interest. Many clustering algorithms are available; the choice of the clustering algorithm can strongly impact the resulting strata because various algorithms focus on different aspects of the observed data.

Bayesian modeling of multivariate loss reserving data based on scale mixtures of multivariate normal distributions: estimation and case influence diagnostics

One of the most important problems in general insurance is estimating the loss reserve distribution. In this article, we develop Bayesian multivariate loss reserving models for cases where losses and random effects are assumed to be distributed under the scale mixtures of multivariate normal (SMMN) distributions. This class of distributions, which contains heavy-tailed multivariate distributions such as student’s t, Pearson type VII, variance-gamma, slash and contaminated normal distributions, can be often used for robust inferences; when the assumptions of normality become questionable. The hierarchical structure of the SMMN representation has the advantage that under a Bayesian paradigm, the parameter estimation is simplified by sampling from multivariate normal distribution using Markov Chain Monte Carlo (MCMC) methods. A Bayesian case deletion influence diagnostics based on q-divergence measures is also presented. Further, simulated and real data sets are analyzed, where we show that the models under the Pearson type VII and the variance-gamma distributions outperform the usual normal models.

Bayesian Nonparametric Functional Mixture Estimation for Time-Series Data, With Application to Estimation of State Employment Totals

The U.S. Bureau of Labor Statistics use monthly, by-state employment totals from the Current Population Survey (CPS) as a key input to develop employment estimates for counties within the states. The monthly CPS by-state totals, however, express high levels of volatility that compromise the accuracy of resulting estimates composed for the counties. Typically-employed models for small area estimation produce de-noised, state-level employment estimates by borrowing information over the survey months, but assume independence among the collection of by-state time series, which is typically violated due to similarities in their underlying economies. We construct Gaussian process and Gaussian Markov random field alternative functional prior specifications, each in a mixture of multivariate Gaussian distributions with a Dirichlet process (DP) mixing measure over the parameters of their covariance or precision matrices. Our DP mixture of functions models allow the data to simultaneously estimate a dependence among the months and between states. A feature of our models is that those functions assigned to the same cluster are drawn from a distribution with the same covariance parameters, so that they are similar, but don't have to be identical. We compare the performances of our two alternatives on synthetic data and apply them to recover de-noised, by-state CPS employment totals for data from $2000-2013$.

Comparing consensus Monte Carlo strategies for distributed Bayesian computation

Consensus Monte Carlo is an algorithm for conducting Monte Carlo based Bayesian inference on large data sets distributed across many worker machines in a data center. The algorithm operates by running a separate Monte Carlo algorithm on each worker machine, which only sees a portion of the full data set. The worker-level posterior samples are then combined to form a Monte Carlo approximation to the full posterior distribution based on the complete data set. We compare several methods of carrying out the combination, including a new method based on approximating worker-level simulations using a mixture of multivariate Gaussian distributions. We find that resampling and kernel density based methods break down after 10 or sometimes fewer dimensions, while the new mixture-based approach works well, but the necessary mixture models take too long to fit.

Stop or Continue Data Collection: A Nonignorable Missing Data Approach for Continuous Variables

Abstract We present an approach to inform decisions about nonresponse follow-up sampling. The basic idea is (i) to create completed samples by imputing nonrespondents’ data under various assumptions about the nonresponse mechanisms, (ii) take hypothetical samples of varying sizes from the completed samples, and (iii) compute and compare measures of accuracy and cost for different proposed sample sizes. As part of the methodology, we present a new approach for generating imputations for multivariate continuous data with nonignorable unit nonresponse. We fit mixtures of multivariate normal distributions to the respondents’ data, and adjust the probabilities of the mixture components to generate nonrespondents’ distributions with desired features. We illustrate the approaches using data from the 2007 U.S. Census of Manufactures.

Quantitative reproducibility analysis for identifying reproducible targets from high-throughput experiments

BackgroundHigh-throughput assays are widely used in biological research to select potential targets. One single high-throughput experiment can efficiently study a large number of candidates simultaneously, but is subject to substantial variability. Therefore it is scientifically important to performance quantitative reproducibility analysis to identify reproducible targets with consistent and significant signals across replicate experiments. A few methods exist, but all have limitations.MethodsIn this paper, we propose a new method for identifying reproducible targets. Considering a Bayesian hierarchical model, we show that the test statistics from replicate experiments follow a mixture of multivariate Gaussian distributions, with the one component with zero-mean representing the irreproducible targets.ResultsA target is thus classified as reproducible or irreproducible based on its posterior probability belonging to the reproducible components. We study the performance of our proposed method using simulations and a real data example.ConclusionThe proposed method is shown to have favorable performance in identifying reproducible targets compared to other methods.

Multiple Imputation of Missing Categorical and Continuous Values via Bayesian Mixture Models With Local Dependence

ABSTRACTWe present a nonparametric Bayesian joint model for multivariate continuous and categorical variables, with the intention of developing a flexible engine for multiple imputation of missing values. The model fuses Dirichlet process mixtures of multinomial distributions for categorical variables with Dirichlet process mixtures of multivariate normal distributions for continuous variables. We incorporate dependence between the continuous and categorical variables by (1) modeling the means of the normal distributions as component-specific functions of the categorical variables and (2) forming distinct mixture components for the categorical and continuous data with probabilities that are linked via a hierarchical model. This structure allows the model to capture complex dependencies between the categorical and continuous data with minimal tuning by the analyst. We apply the model to impute missing values due to item nonresponse in an evaluation of the redesign of the Survey of Income and Program Participation (SIPP). The goal is to compare estimates from a field test with the new design to estimates from selected individuals from a panel collected under the old design. We show that accounting for the missing data changes some conclusions about the comparability of the distributions in the two datasets. We also perform an extensive repeated sampling simulation using similar data from complete cases in an existing SIPP panel, comparing our proposed model to a default application of multiple imputation by chained equations. Imputations based on the proposed model tend to have better repeated sampling properties than the default application of chained equations in this realistic setting. Supplementary materials for this article are available online.

An approach to improve the predictive power of choice-based conjoint analysis

Conjoint analysis continues to be popular with over 18,000 applications each year. Choice-based conjoint (CBC) analysis is currently the most often used method of conjoint analysis accounting for eight-tenths of all conjoint studies. The CBC employs a multinomial logit model with heterogeneous parameters across the population. The most commonly used models of heterogeneity are the Latent Class Model, the single multivariate normal distribution, or a mixture of multivariate normal distributions. A more recent approach to capture heterogeneity is the Dirichlet Process Mixture (DPM) model and its predecessor Dirichlet Process Prior (DPP) model. The alternative models are empirically tested over eleven CBC data sets with varying characteristics. The DPM model provides the best predictive validity (percent of choices correctly predicted) for each of the eleven datasets studied, and provides a significant improvement over extant models of heterogeneity.