Dirichlet Mixture Model Research Articles

Latent Dirichlet mixture model

Abstract Text representation based on latent topic model is seen as a non-Gaussian problem where the observed words and latent topics are multinomial variables and the topic proportionals are Dirichlet variables. Traditional topic model is established by introducing a single Dirichlet prior to characterize the topic proportionals. The words in a text document are represented by a random mixture of semantic topics. However, in real world, a single Dirichlet distribution may not faithfully reflect the variations of topic proportionals estimated from the heterogeneous documents. To address these variations, we propose a new latent variable model where latent topics and their proportionals are learned by incorporating the prior based on Dirichlet mixture model. The resulting latent Dirichlet mixture model (LDMM) is constructed for topic clustering as well as document clustering. Multiple Dirichlets provide a solution to build structural latent variables in learning representation over a variety of topics. This study carries out the inference for LDMM according to the variational Bayes and the collapsed variational Bayes. Such an unsupervised LDMM is further extended to a supervised LDMM for text classification. Experiments on document representation, summarization and classification show the merit of structural prior in LDMM topic models.

DIMM-SC: a Dirichlet mixture model for clustering droplet-based single cell transcriptomic data.

Single cell transcriptome sequencing (scRNA-Seq) has become a revolutionary tool to study cellular and molecular processes at single cell resolution. Among existing technologies, the recently developed droplet-based platform enables efficient parallel processing of thousands of single cells with direct counting of transcript copies using Unique Molecular Identifier (UMI). Despite the technology advances, statistical methods and computational tools are still lacking for analyzing droplet-based scRNA-Seq data. Particularly, model-based approaches for clustering large-scale single cell transcriptomic data are still under-explored. We developed DIMM-SC, a Dirichlet Mixture Model for clustering droplet-based Single Cell transcriptomic data. This approach explicitly models UMI count data from scRNA-Seq experiments and characterizes variations across different cell clusters via a Dirichlet mixture prior. We performed comprehensive simulations to evaluate DIMM-SC and compared it with existing clustering methods such as K-means, CellTree and Seurat. In addition, we analyzed public scRNA-Seq datasets with known cluster labels and in-house scRNA-Seq datasets from a study of systemic sclerosis with prior biological knowledge to benchmark and validate DIMM-SC. Both simulation studies and real data applications demonstrated that overall, DIMM-SC achieves substantially improved clustering accuracy and much lower clustering variability compared to other existing clustering methods. More importantly, as a model-based approach, DIMM-SC is able to quantify the clustering uncertainty for each single cell, facilitating rigorous statistical inference and biological interpretations, which are typically unavailable from existing clustering methods. DIMM-SC has been implemented in a user-friendly R package with a detailed tutorial available on www.pitt.edu/∼wec47/singlecell.html. wei.chen@chp.edu or hum@ccf.org. Supplementary data are available at Bioinformatics online.

A Novel Model-Based Approach for Medical Image Segmentation Using Spatially Constrained Inverted Dirichlet Mixture Models

In this paper, we present a novel statistical approach to medical image segmentation. This approach is based on finite mixture models with spatial smoothness constrains. The main advantages of the proposed approach can be summarized as follows. Firstly, the proposed model is based on inverted Dirichlet mixture models, which have demonstrated better performance in modeling positive data (e.g., images) than Gaussian mixture models. Secondly, we integrate spatial relationships between pixels with the inverted Dirichlet mixture model, which makes it more robust against noise and image contrast levels. Finally, we develop a variational Bayes method to learn the proposed model, such that the model parameters and model complexity (i.e., the number of mixture components) can be estimated simultaneously in a unified framework. The performance of the proposed approach in medical image segmentation is compared with some state-of-the-art segmentation approaches through various numerical experiments on both simulated and real medical images.

UNSUPERVISED LEARNING FOR RIPENESS ESTIMATION FROM GRAPE SEEDS IMAGES

Abstract Estimating the current stage of grape ripeness is a crucial step in wine making and becomes especially important during harvesting. Visual inspection of grape seeds is one method to achieve this goal without performing chemical analysis, however this method is prone to failure. In this paper, we propose an unsupervised visual inspection system for grape ripeness estimation using the Dirichlet Mixture Model (DMM). Experimental analysis using real world data demonstrates that our approach can be used to estimate different ripeness stages from unlabeled grape seeds catalogs.

SDEAP: a splice graph based differential transcript expression analysis tool for population data.

Differential transcript expression (DTE) analysis without predefined conditions is critical to biological studies. For example, it can be used to discover biomarkers to classify cancer samples into previously unknown subtypes such that better diagnosis and therapy methods can be developed for the subtypes. Although several DTE tools for population data, i.e. data without known biological conditions, have been published, these tools either assume binary conditions in the input population or require the number of conditions as a part of the input. Fixing the number of conditions to binary is unrealistic and may distort the results of a DTE analysis. Estimating the correct number of conditions in a population could also be challenging for a routine user. Moreover, the existing tools only provide differential usages of exons, which may be insufficient to interpret the patterns of alternative splicing across samples and restrains the applications of the tools from many biology studies. We propose a novel DTE analysis algorithm, called SDEAP, that estimates the number of conditions directly from the input samples using a Dirichlet mixture model and discovers alternative splicing events using a new graph modular decomposition algorithm. By taking advantage of the above technical improvement, SDEAP was able to outperform the other DTE analysis methods in our extensive experiments on simulated data and real data with qPCR validation. The prediction of SDEAP also allowed us to classify the samples of cancer subtypes and cell-cycle phases more accurately. SDEAP is publicly available for free at https://github.com/ewyang089/SDEAP/wiki CONTACT: yyang027@cs.ucr.edu; jiang@cs.ucr.eduSupplementary information: Supplementary data are available at Bioinformatics online.

A structured Dirichlet mixture model for compositional data: inferential and applicative issues

The flexible Dirichlet (FD) distribution (Ongaro and Migliorati in J. Multvar. Anal. 114: 412---426, 2013) makes it possible to preserve many theoretical properties of the Dirichlet one, without inheriting its lack of flexibility in modeling the various independence concepts appropriate for compositional data, i.e. data representing vectors of proportions. In this paper we tackle the potential of the FD from an inferential and applicative viewpoint. In this regard, the key feature appears to be the special structure defining its Dirichlet mixture representation. This structure determines a simple and clearly interpretable differentiation among mixture components which can capture the main features of a large variety of data sets. Furthermore, it allows a substantially greater flexibility than the Dirichlet, including both unimodality and a varying number of modes. Very importantly, this increased flexibility is obtained without sharing many of the inferential difficulties typical of general mixtures. Indeed, the FD displays the identifiability and likelihood behavior proper to common (non-mixture) models. Moreover, thanks to a novel non random initialization based on the special FD mixture structure, an efficient and sound estimation procedure can be devised which suitably combines EM-types algorithms. Reliable complete-data likelihood-based estimators for standard errors can be provided as well.

Combining regional estimation and historical floods: A multivariate semiparametric peaks‐over‐threshold model with censored data

AbstractThe estimation of extreme flood quantiles is challenging due to the relative scarcity of extreme data compared to typical target return periods. Several approaches have been developed over the years to face this challenge, including regional estimation and the use of historical flood data. This paper investigates the combination of both approaches using a multivariate peaks‐over‐threshold model that allows estimating altogether the intersite dependence structure and the marginal distributions at each site. The joint distribution of extremes at several sites is constructed using a semiparametric Dirichlet Mixture model. The existence of partially missing and censored observations (historical data) is accounted for within a data augmentation scheme. This model is applied to a case study involving four catchments in Southern France, for which historical data are available since 1604. The comparison of marginal estimates from four versions of the model (with or without regionalizing the shape parameter; using or ignoring historical floods) highlights significant differences in terms of return level estimates. Moreover, the availability of historical data on several nearby catchments allows investigating the asymptotic dependence properties of extreme floods. Catchments display a significant amount of asymptotic dependence, calling for adapted multivariate statistical models.

Application of Gibbs sampling in efficient hyperspectral unmixing based on the mixtures of Dirichlet components

The independent component analysis has been commonly employed in hyperspectral unmixing. However, the success of this method is highly dependent on the independency of its sources assumption. Dependent component analysis (DECA) algorithm, which utilizes a Dirichlet mixture model, was developed to provide more adequate spectral unmixing of dependent sources. Estimation of the unknown model parameters using the expectation maximization algorithm in DECA resulted in some insufficiencies. DECAGibbs algorithm is introduced to improve unmixing accuracy by applying the Gibbs sampling method to the parameter estimation process of DECA, which is conducted in different manners of modeling the observations. Functionality of the DECAGibbs algorithm is examined through the artificial datasets and an AVIRIS image of Cuprite, Nevada, indicating better decomposition of mixed observations. Finally, the best performing model was employed in mineralogical mapping of the Lahroud region, northwest Iran, by a Hyperion image. The results represent the high reliability of the proposed method according to the geological studies of the area. Since the practical application of the mixture models relies upon the efficient estimation of their involved parameters, the performance of the DECA algorithm is improved by employing the Bayesian parameter estimation approaches in this research.

Variational Bayesian inference for infinite generalized inverted Dirichlet mixtures with feature selection and its application to clustering

We developed a variational Bayesian learning framework for the infinite generalized Dirichlet mixture model (i.e. a weighted mixture of Dirichlet process priors based on the generalized inverted Di...

A Linear Dirichlet Mixture Model for decomposing scenes: Application to analyzing urban functional zonings

Urban scenes are fundamental to assessing urban landscapes and analyzing the spatial arrangements of functional zonings. Thus, it is important to obtain the information on urban scenes automatically from high-resolution remote sensing images. Previous studies have focused solely on obtaining qualitative labels of scenes (i.e., scene classification) as opposed to quantitative measurements on scenes (i.e., scene decomposition). However, scene classification alone cannot satisfy the demands of practical applications, as most scenes are mixed and composed of many categories. This study aims at 1) developing a decomposition model for quantifying mixed semantics of urban scenes, and 2) applying the proposed model to analyze urban functional zonings. For the first target, a Linear Dirichlet Mixture Model (LDMM) is proposed to quantify the relationships between pure scenes (including one single category) and mixed ones (including multiple categories). Since both mixed and pure scenes are characterized by frequency features (e.g., gray histograms of shapes), the variations in the quantization levels of frequency features could lead to different decomposition results. Accordingly, the stationary Markov process is further introduced to estimate the convergent decomposition results. For the second target, the LDMM was applied to obtain the qualitative categories and their quantitative proportions of urban scenes in Beijing and Zhuhai and to assess their spatial arrangements and temporal changes. Our methods were experimentally verified to be reliable and more effective in performance than the Linear Mixture Model and scene classification techniques. The case studies also suggested that Beijing and Zhuhai significantly differ in urban functional zonings and the functional zonings in Beijing have changed much over time, indicating an important transition in the city's economic structure.

Semi-parametric modeling of excesses above high multivariate thresholds with censored data

How to include censored data in a statistical analysis is a recurrent issue in statistics. In multivariate extremes, the dependence structure of large observations can be characterized in terms of a non parametric angular measure , while marginal excesses above asymptotically large thresholds have a parametric distribution. In this work, a flexible semi-parametric Dirichlet mixture model for angular measures is adapted to the context of censored data and missing components. One major issue is to take into account censoring intervals overlapping the extremal threshold, without knowing whether the corresponding hidden data is actually extreme. Further, the censored likelihood needed for Bayesian inference has no analytic expression. The first issue is tackled using a Poisson process model for extremes, whereas a data augmentation scheme avoids multivariate integration of the Poisson process intensity over both the censored intervals and the failure region above threshold. The implemented MCMC algorithm allows simultaneous estimation of marginal and dependence parameters, so that all sources of uncertainty other than model bias are captured by posterior credible intervals. The method is illustrated on simulated and real data.

Variational learning of hierarchical infinite generalized Dirichlet mixture models and applications

Data clustering is a fundamental unsupervised learning task in several domains such as data mining, computer vision, information retrieval, and pattern recognition. In this paper, we propose and analyze a new clustering approach based on both hierarchical Dirichlet processes and the generalized Dirichlet distribution, which leads to an interesting statistical framework for data analysis and modelling. Our approach can be viewed as a hierarchical extension of the infinite generalized Dirichlet mixture model previously proposed in Bouguila and Ziou (IEEE Trans Neural Netw 21(1):107---122, 2010). The proposed clustering approach tackles the problem of modelling grouped data where observations are organized into groups that we allow to remain statistically linked by sharing mixture components. The resulting clustering model is learned using a principled variational Bayes inference-based algorithm that we have developed. Extensive experiments and simulations, based on two challenging applications namely images categorization and web service intrusion detection, demonstrate our model usefulness and merits.

SIBIS: a Bayesian model for inconsistent protein sequence estimation

The prediction of protein coding genes is a major challenge that depends on the quality of genome sequencing, the accuracy of the model used to elucidate the exonic structure of the genes and the complexity of the gene splicing process leading to different protein variants. As a consequence, today's protein databases contain a huge amount of inconsistency, due to both natural variants and sequence prediction errors. We have developed a new method, called SIBIS, to detect such inconsistencies based on the evolutionary information in multiple sequence alignments. A Bayesian framework, combined with Dirichlet mixture models, is used to estimate the probability of observing specific amino acids and to detect inconsistent or erroneous sequence segments. We evaluated the performance of SIBIS on a reference set of protein sequences with experimentally validated errors and showed that the sensitivity is significantly higher than previous methods, with only a small loss of specificity. We also assessed a large set of human sequences from the UniProt database and found evidence of inconsistency in 48% of the previously uncharacterized sequences. We conclude that the integration of quality control methods like SIBIS in automatic analysis pipelines will be critical for the robust inference of structural, functional and phylogenetic information from these sequences. Source code, implemented in C on a linux system, and the datasets of protein sequences are freely available for download at http://www.lbgi.fr/∼julie/SIBIS.

Sampling from Dirichlet process mixture models with unknown concentration parameter: mixing issues in large data implementations.

We consider the question of Markov chain Monte Carlo sampling from a general stick-breaking Dirichlet process mixture model, with concentration parameter alpha . This paper introduces a Gibbs sampling algorithm that combines the slice sampling approach of Walker (Communications in Statistics - Simulation and Computation 36:45–54, 2007) and the retrospective sampling approach of Papaspiliopoulos and Roberts (Biometrika 95(1):169–186, 2008). Our general algorithm is implemented as efficient open source C++ software, available as an R package, and is based on a blocking strategy similar to that suggested by Papaspiliopoulos (A note on posterior sampling from Dirichlet mixture models, 2008) and implemented by Yau et al. (Journal of the Royal Statistical Society, Series B (Statistical Methodology) 73:37–57, 2011). We discuss the difficulties of achieving good mixing in MCMC samplers of this nature in large data sets and investigate sensitivity to initialisation. We additionally consider the challenges when an additional layer of hierarchy is added such that joint inference is to be made on alpha . We introduce a new label-switching move and compute the marginal partition posterior to help to surmount these difficulties. Our work is illustrated using a profile regression (Molitor et al. Biostatistics 11(3):484–498, 2010) application, where we demonstrate good mixing behaviour for both synthetic and real examples.Electronic supplementary materialThe online version of this article (doi:10.1007/s11222-014-9471-3) contains supplementary material, which is available to authorized users.

Dirichlet mixture modeling to estimate an empirical lower bound for LSF quantization

The line spectral frequencies (LSFs) are commonly used for the linear predictive/autoregressive model in speech and audio coding. Recently, probability density function (PDF)-optimized vector quantization (VQ) has been studied intensively for quantization of LSF parameters. In this paper, we study the VQ performance bound of the LSF parameters. The LSF parameters are transformed to the Δ LSF domain and the underlying distribution of the ΔLSF parameters is modeled by a Dirichlet mixture model (DMM) with a finite number of mixture components. The quantization distortion, in terms of the mean squared error (MSE), is calculated with high rate theory. For LSF quantization, the mapping relation between the perceptually motivated log spectral distortion (LSD) and the MSE is empirically approximated by a polynomial. With this mapping function, the minimum required bit rate (an empirical lower bound) for transparent coding of the LSF under DMM modeling is derived.

Bayesian estimation of Dirichlet mixture model with variational inference

In statistical modeling, parameter estimation is an essential and challengeable task. Estimation of the parameters in the Dirichlet mixture model (DMM) is analytically intractable, due to the integral expressions of the gamma function and its corresponding derivatives. We introduce a Bayesian estimation strategy to estimate the posterior distribution of the parameters in DMM. By assuming the gamma distribution as the prior to each parameter, we approximate both the prior and the posterior distribution of the parameters with a product of several mutually independent gamma distributions. The extended factorized approximation method is applied to introduce a single lower-bound to the variational objective function and an analytically tractable estimation solution is derived. Moreover, there is only one function that is maximized during iterations and, therefore, the convergence of the proposed algorithm is theoretically guaranteed. With synthesized data, the proposed method shows the advantages over the EM-based method and the previously proposed Bayesian estimation method. With two important multimedia signal processing applications, the good performance of the proposed Bayesian estimation method is demonstrated.

Unsupervised clustering and feature weighting based on Generalized Dirichlet mixture modeling

We propose a possibilistic approach for Generalized Dirichlet mixture parameter estimation, data clustering, and feature weighting. The proposed algorithm, called Robust and Unsupervised Learning of Finite Generalized Dirichlet Mixture Models (RULe_GDM), exploits a property of the Generalized Dirichlet distributions that transforms the data to make the features independent and follow Beta distributions. Then, it learns optimal relevance weights for each feature within each cluster. This property makes RULe_GDM suitable for noisy and high-dimensional feature spaces. In addition, RULe_GDM associates two types of memberships with each data sample. The first one is the posterior probability and indicates how well a sample fits each estimated distribution. The second membership represents the degree of typicality and is used to identify and discard noise points and outliers. RULe_GDM minimizes one objective function which combines learning the two membership functions, distribution parameters, and the relevance weights for each feature within each distribution. We also extend our algorithm to find the optimal number of clusters in an unsupervised and efficient way by exploiting some properties of the possibilistic membership function. The performance of RULe_GDM is illustrated and compared to similar algorithms. We use synthetic data to illustrate its robustness to noisy and high dimensional features. We also compare our approach to other relevant algorithms using several standard data sets.

Location Feature Integration for Clustering-Based Speech Separation in Distributed Microphone Arrays

In distributed microphone arrays (DMAs) the source location information can be defined at the intra and inter-node levels. Indeed, while the first type of information results from the diversity of acoustic channels recorded by microphones embedded in the same node, the second is attributed to the differences between the acoustic channels observed by spatially distributed nodes. Both cues are very useful in DMA processing, and the aim of this paper is to utilize both of them to cluster and separate multiple competing speech signals. To capture the intra-node information, we employ the normalized recording vector, while at the inter-node level, we consider different features including the energy level differences with and without the phase differences between nodes. We model the intra-node information using the Watson mixture model (WMM), and propose using the Gamma mixture model (GaMM), Dirichlet mixture model (DMM), and WMM to model different inter-node location features. Furthermore, we propose several integrations of the intra-node and inter-node feature contributions to cluster speech recordings using the expectation maximization algorithm. Finally, simulation results are provided to demonstrate the performance of all ensuing methods.

Robust simultaneous positive data clustering and unsupervised feature selection using generalized inverted Dirichlet mixture models

The discovery, extraction and analysis of knowledge from data rely generally upon the use of unsupervised learning methods, in particular clustering approaches. Much recent research in clustering and data engineering has focused on the consideration of finite mixture models which allow to reason in the face of uncertainty and to learn by example. The adoption of these models becomes a challenging task in the presence of outliers and in the case of high-dimensional data which necessitates the deployment of feature selection techniques. In this paper we tackle simultaneously the problems of cluster validation (i.e. model selection), feature selection and outliers rejection when clustering positive data. The proposed statistical framework is based on the generalized inverted Dirichlet distribution that offers a more practical and flexible alternative to the inverted Dirichlet which has a very restrictive covariance structure. The learning of the parameters of the resulting model is based on the minimization of a message length objective incorporating prior knowledge. We use synthetic data and real data generated from challenging applications, namely visual scenes and objects clustering, to demonstrate the feasibility and advantages of the proposed method.

Variational learning of finite Dirichlet mixture models using component splitting

Finite Dirichlet mixture models have proved to be an effective knowledge representation and inference engine in several machine learning and data mining applications. In this paper, we address the task of learning and selecting finite Dirichlet mixture models in an incremental variational way. A learning algorithm based on component splitting and local model selection is proposed. The merits of the proposed approach are illustrated using synthetic data as well as real challenging applications involving object detection, text documents clustering and distinguishing photographic images from computer graphics.