Infinite Mixture Model Research Articles

Background subtraction using infinite asymmetric Gaussian mixture models with simultaneous feature selection

Mixture models are broadly applied in image processing domains. Related existing challenges include failure to approximate exact data shapes, estimate correct number of components, and ignore irrelevant features. In this study, the authors develop a statistical self-refinement framework for the background subtraction task by using Dirichlet Process-based asymmetric Gaussian mixture model. The parameters of this model are learned using variational inference methods. They also incorporate feature selection simultaneously within the framework to avoid noisy influence from uninformative features. To validate the proposed framework, they report their results on background subtraction tasks on 8 different datasets for infrared and visible videos.

On a class of repulsive mixture models

Finite or infinite mixture models are routinely used in Bayesian statistical practice for tasks such as clustering or density estimation. Such models are very attractive due to their flexibility and tractability. However, a common problem in fitting these or other discrete models to data is that they tend to produce a large number of overlapping clusters. Some attention has been given in the statistical literature to models that include a repulsive feature, i.e., that encourage separation of mixture components. We study here a method that has been shown to achieve this goal without sacrificing flexibility or model fit. The model is a special case of Gibbs measures, with a parameter that controls the level of repulsion that allows construction of d-dimensional probability densities whose coordinates tend to repel each other. This approach was successfully used for density regression in Quinlan et al. (J Stat Comput Simul 88(15):2931–2947, 2018). We detail some of the global properties of the repulsive family of distributions and offer some further insight by means of a small simulation study.

High-resolution temporal profiling of the human gut microbiome reveals consistent and cascading alterations in response to dietary glycans

BackgroundDietary glycans, widely used as food ingredients and not directly digested by humans, are of intense interest for their beneficial roles in human health through shaping the microbiome. Characterizing the consistency and temporal responses of the gut microbiome to glycans is critical for rationally developing and deploying these compounds as therapeutics.MethodsWe investigated the effect of two chemically distinct glycans (fructooligosaccharides and polydextrose) through three clinical studies conducted with 80 healthy volunteers. Stool samples, collected at dense temporal resolution (~ 4 times per week over 10 weeks) and analyzed using shotgun metagenomic sequencing, enabled detailed characterization of participants’ microbiomes. For analyzing the microbiome time-series data, we developed MC-TIMME2 (Microbial Counts Trajectories Infinite Mixture Model Engine 2.0), a purpose-built computational tool based on nonparametric Bayesian methods that infer temporal patterns induced by perturbations and groups of microbes sharing these patterns.ResultsOverall microbiome structure as well as individual taxa showed rapid, consistent, and durable alterations across participants, regardless of compound dose or the order in which glycans were consumed. Significant changes also occurred in the abundances of microbial carbohydrate utilization genes in response to polydextrose, but not in response to fructooligosaccharides. Using MC-TIMME2, we produced detailed, high-resolution temporal maps of the microbiota in response to glycans within and across microbiomes.ConclusionsOur findings indicate that dietary glycans cause reproducible, dynamic, and differential alterations to the community structure of the human microbiome.

Bayesian nonparametric latent class model for longitudinal data

Latent class models have been widely used in longitudinal studies to uncover unobserved heterogeneity in a population and find the characteristics of the latent classes simultaneously using the class allocation probabilities dependent on predictors. However, previous latent class models for longitudinal data suffer from uncertainty in the choice of the number of latent classes. In this study, we propose a Bayesian nonparametric latent class model for longitudinal data, which allows the number of latent classes to be inferred from the data. The proposed model is an infinite mixture model with predictor-dependent class allocation probabilities; an individual longitudinal trajectory is described by the class-specific linear mixed effects model. The model parameters are estimated using Markov chain Monte Carlo methods. The proposed model is validated using a simulated example and a real-data example for characterizing latent classes of estradiol trajectories over the menopausal transition using data from the Study of Women's Health Across the Nation.

Coupled IGMM-GANs with Applications to Anomaly Detection in Human Mobility Data

Detecting anomalous activity in human mobility data has a number of applications, including road hazard sensing, telematics-based insurance, and fraud detection in taxi services and ride sharing. In this article, we address two challenges that arise in the study of anomalous human trajectories: (1) a lack of ground truth data on what defines an anomaly and (2) the dependence of existing methods on significant pre-processing and feature engineering. Although generative adversarial networks (GANs) seem like a natural fit for addressing these challenges, we find that existing GAN-based anomaly detection algorithms perform poorly due to their inability to handle multimodal patterns. For this purpose, we introduce an infinite Gaussian mixture model coupled with (bidirectional) GANs—IGMM-GAN—that is able to generate synthetic, yet realistic, human mobility data and simultaneously facilitates multimodal anomaly detection. Through the estimation of a generative probability density on the space of human trajectories, we are able to generate realistic synthetic datasets that can be used to benchmark existing anomaly detection methods. The estimated multimodal density also allows for a natural definition of outlier that we use for detecting anomalous trajectories. We illustrate our methodology and its improvement over existing GAN anomaly detection on several human mobility datasets, along with MNIST.

Tractable Bayesian density regression via logit stick-breaking priors

There is a growing interest in learning how the distribution of a response variable changes with a set of observed predictors. Bayesian nonparametric dependent mixture models provide a flexible approach to address this goal. However, several formulations require computationally demanding algorithms for posterior inference. Motivated by this issue, we study a class of predictor-dependent infinite mixture models, which relies on a simple representation of the stick-breaking prior via sequential logistic regressions. This formulation maintains the same desirable properties of popular predictor-dependent stick-breaking priors, and leverages a recent Pólya-gamma data augmentation to facilitate the implementation of several computational methods for posterior inference. These routines include Markov chain Monte Carlo via Gibbs sampling, expectation–maximization algorithms, and mean-field variational Bayes for scalable inference, thereby stimulating a wider implementation of Bayesian density regression by practitioners. The algorithms associated with these methods are presented in detail and tested in a toxicology study.

A Bayesian nonparametric model and its application in insurance loss prediction

Predicting insurance losses is an eternal focus of actuarial science in the insurance sector. Due to the existence of complicated features such as skewness, heavy tail, and multi-modality, traditional parametric models are often inadequate to describe the distribution of losses, calling for a mature application of Bayesian methods. In this study we explore a Gaussian mixture model based on Dirichlet process priors. Using three automobile insurance datasets, we employ the probit stick-breaking method to incorporate the effect of covariates into the weight of the mixture component, improve its hierarchical structure, and propose a Bayesian nonparametric model that can identify the unique regression pattern of different samples. Moreover, an advanced updating algorithm of slice sampling is integrated to apply an improved approximation to the infinite mixture model. We compare our framework with four common regression techniques: three generalized linear models and a dependent Dirichlet process ANOVA model. The empirical results show that the proposed framework flexibly characterizes the actual loss distribution in the insurance datasets and demonstrates superior performance in the accuracy of data fitting and extrapolating predictions, thus greatly extending the application of Bayesian methods in the insurance sector.

Interpretability With Accurate Small Models.

Models often need to be constrained to a certain size for them to be considered interpretable. For example, a decision tree of depth 5 is much easier to understand than one of depth 50. Limiting model size, however, often reduces accuracy. We suggest a practical technique that minimizes this trade-off between interpretability and classification accuracy. This enables an arbitrary learning algorithm to produce highly accurate small-sized models. Our technique identifies the training data distribution to learn from that leads to the highest accuracy for a model of a given size. We represent the training distribution as a combination of sampling schemes. Each scheme is defined by a parameterized probability mass function applied to the segmentation produced by a decision tree. An Infinite Mixture Model with Beta components is used to represent a combination of such schemes. The mixture model parameters are learned using Bayesian Optimization. Under simplistic assumptions, we would need to optimize for O(d) variables for a distribution over a d-dimensional input space, which is cumbersome for most real-world data. However, we show that our technique significantly reduces this number to a fixed set of eight variables at the cost of relatively cheap preprocessing. The proposed technique is flexible: it is model-agnostic, i.e., it may be applied to the learning algorithm for any model family, and it admits a general notion of model size. We demonstrate its effectiveness using multiple real-world datasets to construct decision trees, linear probability models and gradient boosted models with different sizes. We observe significant improvements in the F1-score in most instances, exceeding an improvement of 100% in some cases.

Modeling and Clustering Positive Vectors via Nonparametric Mixture Models of Liouville Distributions.

In this article, we propose an effective mixture model-based approach to modeling and clustering positive data vectors. Our mixture model is based on the inverted Beta-Liouville (IBL) distribution which is extracted from the family of Liouville distributions. To cope with the problem of determining the appropriate number of clusters in our approach, a nonparametric Bayesian framework is used to extend the IBL mixture to an infinite mixture model in which the number of clusters is assumed to be infinite initially and will be inferred automatically during the learning process. To optimize the proposed model, we propose a convergence-guaranteed learning algorithm based on the averaged collapsed variational Bayes inference that can effectively learn model parameters with closed-form solutions. The effectiveness of the proposed infinite IBL mixture model for modeling and clustering positive vectors is validated through both synthetic and real-world data sets.

Letter to the Editor

Hutson and Vexler (2018) demonstrate an example of aliasing with the beta and normal distribution. This letter presents another illustration of aliasing using the beta and normal distributions via an infinite mixture model, inspired by the problem of modeling placebo response.

Maximum-Likelihood Parameter Estimation of the Product Model for Multilook Polarimetric SAR Data

The product model is assumed to be an appropriate statistical model for multilook polarimetric synthetic radar data (PolSAR). According to this model, the observed signal is considered as the product of independent random variates of a complex Gaussian speckle and a non-Gaussian texture. With different texture distributions, the product model leads to different expressions for the compound distribution considered as an infinite mixture model. In this paper, the maximum-likeli-hood (ML) estimator is derived to jointly estimate the speckle and texture parameters in the compound distribution model using the multilook polarimetric radar data. In particular, we estimate: 1) the equivalent number of looks; 2) the covariance matrix of the speckle component; and 3) the texture distribution parameters. The expectation–maximization algorithm is developed to compute the ML estimates of the unknown parameters. The hybrid Cramer–Rao bounds (HCRBs) are also derived for these parameters. First, a general HCRB expression is derived under an arbitrary texture distribution. Then, this expression is simplified for a specific texture distribution. The performance of the ML is compared with the performance of other known estimators using the simulated and real multilook PolSAR data. For real data, a goodness of fit of multilook PolSAR data histograms is used to assess the fitting accuracy of the compound distributions using different estimators.

Variational infinite heterogeneous mixture model for semi-supervised clustering of heart enhancers.

MotivationMammalian genomes can contain thousands of enhancers but only a subset are actively driving gene expression in a given cellular context. Integrated genomic datasets can be harnessed to predict active enhancers. One challenge in integration of large genomic datasets is the increasing heterogeneity: continuous, binary and discrete features may all be relevant. Coupled with the typically small numbers of training examples, semi-supervised approaches for heterogeneous data are needed; however, current enhancer prediction methods are not designed to handle heterogeneous data in the semi-supervised paradigm.ResultsWe implemented a Dirichlet Process Heterogeneous Mixture model that infers Gaussian, Bernoulli and Poisson distributions over features. We derived a novel variational inference algorithm to handle semi-supervised learning tasks where certain observations are forced to cluster together. We applied this model to enhancer candidates in mouse heart tissues based on heterogeneous features. We constrained a small number of known active enhancers to appear in the same cluster, and 47 additional regions clustered with them. Many of these are located near heart-specific genes. The model also predicted 1176 active promoters, suggesting that it can discover new enhancers and promoters.Availability and implementationWe created the ‘dphmix’ Python package: https://pypi.org/project/dphmix/.Supplementary information Supplementary data are available at Bioinformatics online.

Fuzzy Transfer Learning Using an Infinite Gaussian Mixture Model and Active Learning

Transfer learning is gaining considerable attention due to its ability to leverage previously acquired knowledge to assist in completing a prediction task in a related domain. Fuzzy transfer learning, which is based on fuzzy system (especially fuzzy rule-based models), has been developed because of its capability to deal with the uncertainty in transfer learning. However, two issues with fuzzy transfer learning have not yet been resolved: choosing an appropriate source domain and efficiently selecting labeled data for the target domain. This paper proposes an innovative method based on fuzzy rules that combines an infinite Gaussian mixture model (IGMM) with active learning to enhance the performance and generalizability of the constructed model. An IGMM is used to identify the data structures in the source and target domains providing a promising solution to the domain selection dilemma. Further, we exploit the interactive query strategy in active learning to correct imbalances in the knowledge to improve the generalizability of fuzzy learning models. Through experiments on synthetic datasets, we demonstrate the rationality of employing an IGMM and the effectiveness of applying an active learning technique. Additional experiments on real-world datasets further support the capabilities of the proposed method in practical situations.

Modelling bimodality of length of tourist stay

Empirically, the length of stay by tourists at their destination usually presents bimodality, overdispersion and non-zero observations, and classical distributions do not seem to fit this type of data very appropriately. In this paper, we introduce two distributions which accommodate bimodality. One is a flexible discrete distribution which can be applied to both bimodal and unimodal data sets. The second distribution is an infinite mixture model that accounts for unobserved heterogeneity in the mean parameter, thus reflecting the heterogeneous preferences of tourists. Both models are suitable for the inclusion of covariates. Our empirical results show that each of these models is suitable and provides a reasonably good fit. Of the two, the infinite mixture model is preferred.

Audience segmentation to improve targeting of conservation interventions for hunters.

Audience segmentation could help improve the effectiveness of conservation interventions. Marketers use audience segmentation to define the target audience of a campaign. The technique involves subdividing a general population into groups that share similar profiles, such as sociodemographic or behavioral characteristics. Interventions are then designed to target the group or groups of interest. We explored the potential of audience segmentation for use in defining conservation target groups with a case study of hunters in Liberia. Using 2 data sets describing households (n = 476) and hunters (n = 205), we applied a clustering method in which infinite binomial mixture models group hunters and households according to livelihood and behavior variables and a simple method to define target groups based on hunting impact (hunting households and high-impact hunters). Clusters of hunters and households differed in their experiences with confiscation of catch at roadblocks and participation in livelihood-support programs, indicating that these interventions operate unevenly across subsets of the population. By contrast, the simple method masked these insights because profiles of hunting households and high-impact hunters were similar to those of the general population. Clustering results could be used to guide the development of livelihood and regulatory interventions. For example, a commonly promoted agricultural activity, cocoa farming, was practiced by only 2% (out of 87) of the largest hunter cluster of nonlocal gun hunters but was prevalent among local trappers, suggesting that assistance aimed at cocoa farmers is less appropriate for the former group. Our results support the use of audience segmentation across multiple variables to improve targeted intervention designs in conservation.

Modeling the Variability of Active Galactic Nuclei by an Infinite Mixture of Ornstein–Uhlenbeck (OU) Processes

Abstract We develop an infinite mixture model of Ornstein–Uhlenbeck (OU) processes for describing the optical variability of QSOs based on treating the variability as a stochastic process. This enables us to get the parameters of the power spectral densities (PSDs) on their brightness variations by providing more flexible description of PSDs than the models based on a single OU process (damped random walk (DRW). We apply this model to 67,507 variable objects extracted from Sloan Digital Sky Survey (SDSS) Stripe 82 photometric data and succeed in showing very high precision in identifying QSOs (∼99% levels in completeness and purity) among variable objects based only on their variability, by investigating on 9855 spectroscopically confirmed objects (7714 QSOs and 2141 stars) in the data of SDSS Data Release 12 (DR12), with sufficient and accurate multiple measurements of their brightness. By comparing our results with the values based on other models that are used in previous research, it is revealed that our model can be used as the most effective method for selecting QSOs from a variable object catalog, especially regarding completeness and purity. The main reason for improved identification rates is the ability of our model to separate clearly QSOs and stars, especially on the small fraction of QSOs with variabilities that can be described better than with the simple DRW model.

A coupled ETAS-I2GMM point process with applications to seismic fault detection

Epidemic-type aftershock sequence (ETAS) point process is a common model for the occurrence of earthquake events. The ETAS model consists of a stationary background Poisson process modeling spontaneous earthquakes and a triggering kernel representing the space–time-magnitude distribution of aftershocks. Popular nonparametric methods for estimation of the background intensity include histograms and kernel density estimators. While these methods are able to capture local spatial heterogeneity in the intensity of spontaneous events, they do not capture well patterns resulting from fault line structure over larger spatial scales. Here we propose a two-layer infinite Gaussian mixture model for clustering of earthquake events into fault-like groups over intermediate spatial scales. We introduce a Monte Carlo expectation-maximization (EM) algorithm for joint inference of the ETAS-I2GMM model and then apply the model to the Southern California Earthquake Catalog. We illustrate the advantages of the ETAS-I2GMM model in terms of both goodness of fit of the intensity and recovery of fault line clusters in the Community Fault Model 3.0 from earthquake occurrence data.

Reconciling differences in natural tags to infer demographic and genetic connectivity in marine fish populations

Processes regulating population connectivity are complex, ranging from extrinsic environmental factors to intrinsic individual based features, and are a major force shaping the persistence of fish species and population responses to harvesting and environmental change. Here we developed an integrated assessment of demographic and genetic connectivity of European flounder Platichthys flesus in the northeast Atlantic (from the Norwegian to the Portuguese coast) and Baltic Sea. Specifically, we used a Bayesian infinite mixture model to infer the most likely number of natal sources of individuals based on otolith near core chemical composition. Simultaneously, we characterised genetic connectivity via microsatellite DNA markers, and evaluated how the combined use of natural tags informed individual movement and long-term population exchange rates. Individual markers provided different insights on movement, with otolith chemistry delineating Norwegian and Baltic Sea sources, whilst genetic markers showed a latitudinal pattern which distinguished southern peripheral populations along the Iberian coast. Overall, the integrated use of natural tags resulted in outcomes that were not readily anticipated by individual movement or gene flow markers alone. Our ecological and evolutionary approach provided a synergistic view on connectivity, which will be paramount to align biological and management units and safeguard species’ biocomplexity.

Variational Bayesian Learning for Dirichlet Process Mixture of Inverted Dirichlet Distributions in Non-Gaussian Image Feature Modeling.

In this paper, we develop a novel variational Bayesian learning method for the Dirichlet process (DP) mixture of the inverted Dirichlet distributions, which has been shown to be very flexible for modeling vectors with positive elements. The recently proposed extended variational inference (EVI) framework is adopted to derive an analytically tractable solution. The convergency of the proposed algorithm is theoretically guaranteed by introducing single lower bound approximation to the original objective function in the EVI framework. In principle, the proposed model can be viewed as an infinite inverted Dirichlet mixture model that allows the automatic determination of the number of mixture components from data. Therefore, the problem of predetermining the optimal number of mixing components has been overcome. Moreover, the problems of overfitting and underfitting are avoided by the Bayesian estimation approach. Compared with several recently proposed DP-related methods and conventional applied methods, the good performance and effectiveness of the proposed method have been demonstrated with both synthesized data and real data evaluations.

A Tutorial on Levels of Granularity: From Histograms to Clusters to Predictive Distributions

Consider the problem of modeling datasets such as numbers of accidents in a population of insured persons, or incidences of an illness in a population. Various levels of detail or granularity may be considered in describing the parent population. The levels used in fitting data and hence in describing the population may vary from a single distribution, possibly with extreme values, to a bimodal distribution, to a mixture of two or more distributions via the Finite Mixture Model, to modeling the population at the individual level via a compound model, which may be viewed as an infinite mixture model. Given a dataset, it is shown how to evaluate the fits of the various models by information criteria. Two datasets are considered in detail, one discrete, the other, continuous.