Infinite Mixture Research Articles

Bounded Asymmetrical Student's-t Mixture Model

The finite mixture model based on the Student's-t distribution, which is heavily tailed and more robust than the Gaussian mixture model (GMM), is a flexible and powerful tool to address many computer vision and pattern recognition problems. However, the Student's-t distribution is unbounded and symmetrical around its mean. In many applications, the observed data are digitalized and have bounded support. The distribution of the observed data usually has an asymmetric form. A new finite bounded asymmetrical Student's-t mixture model (BASMM), which includes the GMM and the Student's-t mixture model (SMM) as special cases, is presented in this paper. We propose an extension of the Student's-t distribution in this paper. This new distribution is sufficiently flexible to fit different shapes of observed data, such as non-Gaussian, nonsymmetric, and bounded support data. Another advantage of the proposed model is that each of its components can model the observed data with different bounded support regions. In order to estimate the model parameters, previous models represent the Student's-t distributions as an infinite mixture of scaled Gaussians. We propose an alternate approach in order to minimize the higher bound on the data negative log-likelihood function, and directly deal with the Student's-t distribution. As an application, our method has been applied to image segmentation with promising results.

Bayesian nonparametric estimation of a copula

A copula can fully characterize the dependence of multiple variables. The purpose of this paper is to provide a Bayesian nonparametric approach to the estimation of a copula, and we do this by mixing over a class of parametric copulas. In particular, we show that any bivariate copula density can be arbitrarily accurately approximated by an infinite mixture of Gaussian copula density functions. The model can be estimated by Markov Chain Monte Carlo methods and the model is demonstrated on both simulated and real data sets.

Bayesian Nonparametric Inference for a Multivariate Copula Function

The paper presents a general Bayesian nonparametric approach for estimating a high dimensional copula. We first introduce the skew–normal copula, which we then extend to an infinite mixture model. The skew–normal copula fixes some limitations in the Gaussian copula. An MCMC algorithm is developed to draw samples from the correct posterior distribution and the model is investigated using both simulated and real applications.

A new sufficient condition for identifiability of countably infinite mixtures

While identifiability of finite mixtures for a wide range of distributions has been studied by statisticians for decades, discussion on countably infinite mixtures is still limited. This article provides an sufficient condition by means of well-ordered sets and uniform convergence of series. It is then applied to revisit some examples for which the identifiability is well established and then explore the identifiability for several distribution families, including normal, gamma, Cauchy, noncentral \(\chi ^2\), multivariate normal distributions.

Nonparametric Bayesian modeling of complex networks: an introduction

Modeling structure in complex networks using Bayesian nonparametrics makes it possible to specify flexible model structures and infer the adequate model complexity from the observed data. This article provides a gentle introduction to nonparametric Bayesian modeling of complex networks: Using an infinite mixture model as running example, we go through the steps of deriving the model as an infinite limit of a finite parametric model, inferring the model parameters by Markov chain Monte Carlo, and checking the model?s fit and predictive performance. We explain how advanced nonparametric models for complex networks can be derived and point out relevant literature.

Bayesian semiparametric multivariate GARCH modeling

This paper proposes a Bayesian nonparametric modeling approach for the return distribution in multivariate GARCH models. In contrast to the parametric literature the return distribution can display general forms of asymmetry and thick tails. An infinite mixture of multivariate normals is given a flexible Dirichlet process prior. The GARCH functional form enters into each of the components of this mixture. We discuss conjugate methods that allow for scale mixtures and nonconjugate methods which provide mixing over both the location and scale of the normal components. MCMC methods are introduced for posterior simulation and computation of the predictive density. Bayes factors and density forecasts with comparisons to GARCH models with Student-t innovations demonstrate the gains from our flexible modeling approach.

PhyloBayes MPI: Phylogenetic Reconstruction with Infinite Mixtures of Profiles in a Parallel Environment

Modeling across site variation of the substitution process is increasingly recognized as important for obtaining more accurate phylogenetic reconstructions. Both finite and infinite mixture models have been proposed and have been shown to significantly improve on classical single-matrix models. Compared with their finite counterparts, infinite mixtures have a greater expressivity. However, they are computationally more challenging. This has resulted in practical compromises in the design of infinite mixture models. In particular, a fast but simplified version of a Dirichlet process model over equilibrium frequency profiles implemented in PhyloBayes has often been used in recent phylogenomics studies, while more refined model structures, more realistic and empirically more fit, have been practically out of reach. We introduce a message passing interface version of PhyloBayes, implementing the Dirichlet process mixture models as well as more classical empirical matrices and finite mixtures. The parallelization is made efficient thanks to the combination of two algorithmic strategies: a partial Gibbs sampling update of the tree topology and the use of a truncated stick-breaking representation for the Dirichlet process prior. The implementation shows close to linear gains in computational speed for up to 64 cores, thus allowing faster phylogenetic reconstruction under complex mixture models. PhyloBayes MPI is freely available from our website www.phylobayes.org.

Interpreting meta‐regression: application to recent controversies in antidepressants’ efficacy

A recent meta-regression of antidepressant efficacy on baseline depression severity has caused considerable controversy in the popular media. A central source of the controversy is a lack of clarity about the relation of meta-regression parameters to corresponding parameters in models for subject-level data. This paper focuses on a linear regression with continuous outcome and predictor, a case that is often considered less problematic. We frame meta-regression in a general mixture setting that encompasses both finite and infinite mixture models. In many applications of meta-analysis, the goal is to evaluate the efficacy of a treatment from several studies, and authors use meta-regression on grouped data to explain variations in the treatment efficacy by study features. When the study feature is a characteristic that has been averaged over subjects, it is difficult not to interpret the meta-regression results on a subject level, a practice that is still widespread in medical research. Although much of the attention in the literature is on methods of estimating meta-regression model parameters, our results illustrate that estimation methods cannot protect against erroneous interpretations of meta-regression on grouped data. We derive relations between meta-regression parameters and within-study model parameters and show that the conditions under which slopes from these models are equal cannot be verified on the basis of group-level information only. The effects of these model violations cannot be known without subject-level data. We conclude that interpretations of meta-regression results are highly problematic when the predictor is a subject-level characteristic that has been averaged over study subjects.

Convergence of latent mixing measures in finite and infinite mixture models

This paper studies convergence behavior of latent mixing measures that arise in finite and infinite mixture models, using transportation distances (i.e., Wasserstein metrics). The relationship between Wasserstein distances on the space of mixing measures and f-divergence functionals such as Hellinger and Kullback-Leibler distances on the space of mixture distributions is investigated in detail using various identifiability conditions. Convergence in Wasserstein metrics for discrete measures implies convergence of individual atoms that provide support for the measures, thereby providing a natural interpretation of convergence of clusters in clustering applications where mixture models are typically employed. Convergence rates of posterior distributions for latent mixing measures are established, for both finite mixtures of multivariate distributions and infinite mixtures based on the Dirichlet process.

Bayesian Semi-Parametric and Non-Parametric Methods in Marketing and Micro-Econometrics

I review and develop Bayesian non-parametric and semi-parametric methods based on finite and infinite mixtures of normals. Applications include regression, IV methods, and random coefficient models.

The beta Weibull-geometric distribution

A new five-parameter distribution called the beta Weibull-geometric (BWG) distribution is proposed. The new distribution is generated from the logit of a beta random variable and includes the Weibull-geometric distribution of Barreto-Souza et al. [The Weibull-geometric distribution, J. Stat. Comput. Simul. 81 (2011), pp. 645–657], beta Weibull (BW), beta exponential, exponentiated Weibull, and some other lifetime distributions as special cases. A comprehensive mathematical treatment of this distribution is provided. The density function can be expressed as an infinite mixture of BW densities and then we derive some mathematical properties of the new distribution from the corresponding properties of the BW distribution. The density function of the order statistics and also estimation of the stress–strength parameter are obtained using two general expressions. To estimate the model parameters, we use the maximum likelihood method and the asymptotic distribution of the estimators is also discussed. The capacity of the new distribution are examined by various tools, using two real data sets.

PhyloBayes MPI: Phylogenetic Reconstruction with Infinite Mixtures of Profiles in a Parallel Environment

PhyloBayes MPI: Phylogenetic Reconstruction with Infinite Mixtures of Profiles in a Parallel Environment

Robust Bayesian Clustering for Replicated Gene Expression Data

Experimental scientific data sets, especially biology data, usually contain replicated measurements. The replicated measurements for the same object are correlated, and this correlation must be carefully dealt with in scientific analysis. In this paper, we propose a robust Bayesian mixture model for clustering data sets with replicated measurements. The model aims not only to accurately cluster the data points taking the replicated measurements into consideration, but also to find the outliers (i.e., scattered objects) which are possibly required to be studied further. A tree-structured variational Bayes (VB) algorithm is developed to carry out model fitting. Experimental studies showed that our model compares favorably with the infinite Gaussian mixture model, while maintaining computational simplicity. We demonstrate the benefits of including the replicated measurements in the model, in terms of improved outlier detection rates in varying measurement uncertainty conditions. Finally, we apply the approach to clustering biological transcriptomics mRNA expression data sets with replicated measurements.

Bayesian Semiparametric Multivariate GARCH Modeling

This paper proposes a Bayesian nonparametric modeling approach for the return distribution in multivariate GARCH models. In contrast to the parametric literature, the return distribution can display general forms of asymmetry and thick tails. An infinite mixture of multivariate normals is given a flexible Dirichlet process prior. The GARCH functional form enters into each of the components of this mixture. We discuss conjugate methods that allow for scale mixtures and nonconjugate methods, which provide mixing over both the location and scale of the normal components. MCMC methods are introduced for posterior simulation and computation of the predictive density. Bayes factors and density forecasts with comparisons to GARCH models with Student-t innovations demonstrate the gains from our flexible modeling approach.

Variational learning for Dirichlet process mixtures of Dirichlet distributions and applications

In this paper, we propose a Bayesian nonparametric approach for modeling and selection based on a mixture of Dirichlet processes with Dirichlet distributions, which can also be seen as an infinite Dirichlet mixture model. The proposed model uses a stick-breaking representation and is learned by a variational inference method. Due to the nature of Bayesian nonparametric approach, the problems of overfitting and underfitting are prevented. Moreover, the obstacle of estimating the correct number of clusters is sidestepped by assuming an infinite number of clusters. Compared to other approximation techniques, such as Markov chain Monte Carlo (MCMC), which require high computational cost and whose convergence is difficult to diagnose, the whole inference process in the proposed variational learning framework is analytically tractable with closed-form solutions. Additionally, the proposed infinite Dirichlet mixture model with variational learning requires only a modest amount of computational power which makes it suitable to large applications. The effectiveness of our model is experimentally investigated through both synthetic data sets and challenging real-life multimedia applications namely image spam filtering and human action videos categorization.

A Family of Near-Exact Distributions Based on Truncations of the Exact Distribution for the Generalized Wilks Lambda Statistic

For the case where at least two sets have an odd number of variables we do not have the exact distribution of the generalized Wilks Lambda statistic in a manageable form, adequate for manipulation. In this article, we develop a family of very accurate near-exact distributions for this statistic for the case where two or three sets have an odd number of variables. We first express the exact characteristic function of the logarithm of the statistic in the form of the characteristic function of an infinite mixture of Generalized Integer Gamma distributions. Then, based on truncations of this exact characteristic function, we obtain a family of near-exact distributions, which, by construction, match the first two exact moments. These near-exact distributions display an asymptotic behaviour for increasing number of variables involved. The corresponding cumulative distribution functions are obtained in a concise and manageable form, relatively easy to implement computationally, allowing for the computation of virtually exact quantiles. We undertake a comparative study for small sample sizes, using two proximity measures based on the Berry-Esseen bounds, to assess the performance of the near-exact distributions for different numbers of sets of variables and different numbers of variables in each set.

The infinite Student's t-factor mixture analyzer for robust clustering and classification

Recently, the Student's t-factor mixture analyzer (tFMA) has been proposed. Compared with the mixture of Student's t-factor analyzers (MtFA), the tFMA has better performance when processing high-dimensional data. Moreover, the factors estimated by the tFMA can be visualized in a low-dimensional latent space, which is not shared by the MtFA. However, as the tFMA belongs to finite mixtures and the related parameter estimation method is based on the maximum likelihood criterion, it could not automatically determine the appropriate model complexity according to the observed data, leading to overfitting. In this paper, we propose an infinite Student's t-factor mixture analyzer (itFMA) to handle this issue. The itFMA is based on the nonparametric Bayesian statistics which assumes infinite number of mixing components in advance, and automatically determines the proper number of components after observing the high-dimensional data. Moreover, we derive an efficient variational inference algorithm for the itFMA. The proposed itFMA and the related variational inference algorithm are used to cluster and classify high-dimensional data. Experimental results of some applications show that the itFMA has good generalization capacity, offering a more robust and powerful performance than other competing approaches.

Estimating a Semiparametric Asymmetric Stochastic Volatility Model with a Dirichlet Process Mixture

In this paper, we extend the parametric, asymmetric, stochastic volatility model (ASV), where returns are correlated with volatility, by flexibly modeling the bivariate distribution of the return and volatility innovations nonparametrically. Its novelty is in modeling the joint, conditional, return-volatility distribution with an infinite mixture of bivariate Normal distributions with mean zero vectors, but having unknown mixture weights and covariance matrices. This semiparametric ASV model nests stochastic volatility models whose innovations are distributed as either Normal or Student-t distributions, plus the response in volatility to unexpected return shocks is more general than the fixed asymmetric response with the ASV model. The unknown mixture parameters are modeled with a Dirichlet process prior. This prior ensures a parsimonious, finite, posterior mixture that best represents the distribution of the innovations and a straightforward sampler of the conditional posteriors. We develop a Bayesian Markov chain Monte Carlo sampler to fully characterize the parametric and distributional uncertainty. Nested model comparisons and out-of-sample predictions with the cumulative marginal-likelihoods, and one-day-ahead, predictive log-Bayes factors between the semiparametric and parametric versions of the ASV model shows the semiparametric model projecting more accurate empirical market returns. A major reason is how volatility responds to an unexpected market movement. When the market is tranquil, expected volatility reacts to a negative (positive) price shock by rising (initially declining, but then rising when the positive shock is large). However, when the market is volatile, the degree of asymmetry and the size of the response in expected volatility is muted. In other words, when times are good, no news is good news, but when times are bad, neither good nor bad news matters with regards to volatility.

On Identifying Primary User Emulation Attacks in Cognitive Radio Systems Using Nonparametric Bayesian Classification

Primary user emulation (PUE) attacks, where attackers mimic the signals of primary users (PUs), can cause significant performance degradation in cognitive radio (CR) systems. Detection of the presence of PUE attackers is thus an important problem. In this paper, using device-specific features, we propose a passive, nonparametric classification method to determine the number of transmitting devices in the PU spectrum. Our method, called DECLOAK, is passive since the sensing device listens and captures signals without injecting any signal to the wireless environment. It is nonparametric because the number of active devices needs not to be known as a priori. Channel independent features are selected forming fingerprints for devices, which cannot be altered postproduction. The infinite Gaussian mixture model (IGMM) is adopted and a modified collapsed Gibbs sampling method is proposed to classify the extracted fingerprints. Due to its unsupervised nature, there is no need to collect legitimate PU fingerprints. In combination with received power and device MAC address, we show through simulation studies that the proposed method can efficiently detect the PUE attack. The performance of DECLOAK is also shown to be superior than that of the classical non-parametric mean shift (MS) based clustering method.

A Nonparametric Bayesian Multipitch Analyzer Based on Infinite Latent Harmonic Allocation

The statistical multipitch analyzer described in this paper estimates multiple fundamental frequencies (F0s) in polyphonic music audio signals produced by pitched instruments. It is based on hierarchic4al nonparametric Bayesian models that can deal with uncertainty of unknown random variables such as model complexities (e.g., the number of F0s and the number of harmonic partials), model parameters (e.g., the values of F0s and the relative weights of harmonic partials), and hyperparameters (i.e., prior knowledge on complexities and parameters). Using these models, we propose a statistical method called infinite latent harmonic allocation (iLHA). To avoid model-complexity control, we allow the observed spectra to contain an unbounded number of sound sources (F0s), each of which is allowed to contain an unbounded number of harmonic partials. More specifically, to model a set of time-sliced spectra, we formulated nested infinite Gaussian mixture models based on hierarchical and generalized Dirichlet processes. To avoid manual tuning of influential hyperparameters, we put noninformative hyperprior distributions on them in a hierarchical manner. For efficient Bayesian inference, we used a modern technique called collapsed variational Bayes. In comparative experiments using audio recordings of piano and guitar solo performances, iLHA yielded promising results and we found that there would be room for improvement based on modeling of temporal continuity and spectral smoothness.