Inverted Dirichlet Mixture Research Articles

Dirichlet Process Mixture of Generalized Inverted Dirichlet Distributions for Positive Vector Data With Extended Variational Inference.

A Bayesian nonparametric approach for estimation of a Dirichlet process (DP) mixture of generalized inverted Dirichlet distributions [i.e., an infinite generalized inverted Dirichlet mixture model (InGIDMM)] has been proposed. The generalized inverted Dirichlet distribution has been proven to be efficient in modeling the vectors that contain only positive elements. Under the classical variational inference (VI) framework, the key challenge in the Bayesian estimation of InGIDMM is that the expectation of the joint distribution of data and variables cannot be explicitly calculated. Therefore, numerical methods are usually applied to simulate the optimal posterior distributions. With the recently proposed extended VI (EVI) framework, we introduce lower bound approximations to the original variational objective function in the VI framework such that an analytically tractable solution can be derived. Hence, the problem in numerical simulation has been overcome. By applying the DP mixture technique, an InGIDMM can automatically determine the number of mixture components from the observed data. Moreover, the DP mixture model with an infinite number of mixture components also avoids the problems of underfitting and overfitting. The performance of the proposed approach is demonstrated with both synthesized data and real-life data applications.

Unsupervised Learning Using Variational Inference on Finite Inverted Dirichlet Mixture Models with Component Splitting

Unsupervised learning has been one of the essentials of pattern recognition and data mining. The role of Dirichlet family of mixture models in this field is inevitable. In this article, we propose a finite Inverted Dirichlet mixture model for unsupervised learning using variational inference. In particular, we develop an incremental algorithm with a component splitting approach for local model selection, which makes the clustering algorithm more efficient. We illustrate our model and learning algorithm with synthetic data and some real applications for occupancy estimation in smart homes and topic learning in images and videos. Extensive comparisons with comparable recent approaches have shown the merits of our proposed model.

A novel approach for modeling positive vectors with inverted Dirichlet-based hidden Markov models

Hidden Markov Models (HMMs) are among the most remarkably powerful probabilistic models, that although been acknowledged for decades have recently made a huge resurgence in the machine learning field. Their ever-growing use to model diversified and heterogeneous data (image,video, audio, time series) in numerous important practical situations is the subject of all forms of perpetual extensions. This work presents what we believe to be the first integration of the Inverted Dirichlet (ID) Mixture Models into the framework of HMMs. The proposed method uses the inverted Dirichlet mixtures to model the emission probabilities also known as observation probabilities. This extension (IDHMM), is motivated by the proven capacity of these mixtures to deal with positive vectors and overcome mixture models’ capability to take into account any ordering or temporal constraints relative to information. The complete inference and parameter estimation are detailed in this work. Applications in the context of image categorization and indoor occupancy detection demonstrate higher performance compared to the extensively used Gaussian mixture-based Hidden Markov Model (GHMM) and the Dirichlet mixture-based hidden Markov Model (DHMM).

Positive Sequential Data Modeling Using Continuous Hidden Markov Models Based on Inverted Dirichlet Mixtures

The hidden Markov model (HMM) has long been one of the most commonly used probability graph models for modeling sequential or time series data. It has been widely used in many fields ranging from speech recognition, face recognition, anomaly detection, to gene function prediction. In this paper, we theoretically propose a variant of the continuous HMM for modeling positive sequential data which are naturally generated in many real-life applications. In contrast with conventional HMMs which often use Gaussian distributions or Gaussian mixture models as the emission probability density, we adopt the inverted Dirichlet mixture model as the emission density to build the HMM. The consideration of inverted Dirichlet mixture model in our case is motivated by its superior modeling capability over Gaussian mixture models for modeling positive data according to several recent studies. In addition, we develop a convergence-guaranteed approach to learning the proposed inverted Dirichlet-based HMM through variational Bayes inference. The effectiveness of the proposed HMM is validated through both synthetic data sets and a real-world application regarding anomaly network intrusion detection. Based on the experimental results, the proposed inverted Dirichlet-based HMM is able to achieve the detection accuracy rates that are about 4%~9% higher than those ones obtained by the compared approaches.

Variational Bayesian Learning for Dirichlet Process Mixture of Inverted Dirichlet Distributions in Non-Gaussian Image Feature Modeling.

In this paper, we develop a novel variational Bayesian learning method for the Dirichlet process (DP) mixture of the inverted Dirichlet distributions, which has been shown to be very flexible for modeling vectors with positive elements. The recently proposed extended variational inference (EVI) framework is adopted to derive an analytically tractable solution. The convergency of the proposed algorithm is theoretically guaranteed by introducing single lower bound approximation to the original objective function in the EVI framework. In principle, the proposed model can be viewed as an infinite inverted Dirichlet mixture model that allows the automatic determination of the number of mixture components from data. Therefore, the problem of predetermining the optimal number of mixing components has been overcome. Moreover, the problems of overfitting and underfitting are avoided by the Bayesian estimation approach. Compared with several recently proposed DP-related methods and conventional applied methods, the good performance and effectiveness of the proposed method have been demonstrated with both synthesized data and real data evaluations.

Bayesian inference by reversible jump MCMC for clustering based on finite generalized inverted Dirichlet mixtures

The goal of constructing models from examples has been approached from different perspectives. Statistical methods have been widely used and proved effective in generating accurate models. Finite Gaussian mixture models have been widely used to describe a wide variety of random phenomena and have played a prominent role in many attempts to develop expressive statistical models in machine learning. However, their effectiveness is limited to applications where underlying modeling assumptions (e.g., the per-components densities are Gaussian) are reasonably satisfied. Thus, much research efforts have been devoted to developing better alternatives. In this paper, we focus on constructing statistical models from positive vectors (i.e., vectors whose elements are strictly greater than zero) for which the generalized inverted Dirichlet (GID) mixture has been shown to be a flexible and powerful parametric framework. In particular, we propose a Bayesian density estimation method based upon mixtures of GIDs. The consideration of Bayesian learning is interesting in several respects. It allows to take uncertainty into account by introducing prior information about the parameters, it allows simultaneous parameters estimation and model selection, and it allows to overcome learning problems related to over- or under-fitting. Indeed, we develop a reversible jump Markov Chain Monte Carlo sampler for GID mixtures that we apply for simultaneous clustering and feature selection in the context of some challenging real-world applications concerning scene classification, action recognition, and video forgery detection.

Variational Bayesian Inference for Finite Inverted Dirichlet Mixture Model and Its Application to Object Detection

As a variant of Finite mixture model (FMM), finite Inverted Dirichlet mixture model (IDMM) can not avoid the conventional challenges, such as how to select the appropriate number of mixture components based on the observed data. Towards easing these issues, we propose a variational inference framework for learning IDMM which has been proved to be an efficient tool for modeling vectors with positive elements. Compared with the conventional Expectation maximization (EM) algorithm commonly used for learning FMM, the proposed approach prevents over-fitting well. Furthermore, it is able to do automatic determination of the number of mixture components and parameters estimation, simultaneously. Experimental results on both synthetic and real data of object detection confirm significant improvements on flexibility and efficiency being achieved.

A Novel Model-Based Approach for Medical Image Segmentation Using Spatially Constrained Inverted Dirichlet Mixture Models

In this paper, we present a novel statistical approach to medical image segmentation. This approach is based on finite mixture models with spatial smoothness constrains. The main advantages of the proposed approach can be summarized as follows. Firstly, the proposed model is based on inverted Dirichlet mixture models, which have demonstrated better performance in modeling positive data (e.g., images) than Gaussian mixture models. Secondly, we integrate spatial relationships between pixels with the inverted Dirichlet mixture model, which makes it more robust against noise and image contrast levels. Finally, we develop a variational Bayes method to learn the proposed model, such that the model parameters and model complexity (i.e., the number of mixture components) can be estimated simultaneously in a unified framework. The performance of the proposed approach in medical image segmentation is compared with some state-of-the-art segmentation approaches through various numerical experiments on both simulated and real medical images.

Variational Bayesian inference for infinite generalized inverted Dirichlet mixtures with feature selection and its application to clustering

We developed a variational Bayesian learning framework for the infinite generalized Dirichlet mixture model (i.e. a weighted mixture of Dirichlet process priors based on the generalized inverted Di...

Robust simultaneous positive data clustering and unsupervised feature selection using generalized inverted Dirichlet mixture models

The discovery, extraction and analysis of knowledge from data rely generally upon the use of unsupervised learning methods, in particular clustering approaches. Much recent research in clustering and data engineering has focused on the consideration of finite mixture models which allow to reason in the face of uncertainty and to learn by example. The adoption of these models becomes a challenging task in the presence of outliers and in the case of high-dimensional data which necessitates the deployment of feature selection techniques. In this paper we tackle simultaneously the problems of cluster validation (i.e. model selection), feature selection and outliers rejection when clustering positive data. The proposed statistical framework is based on the generalized inverted Dirichlet distribution that offers a more practical and flexible alternative to the inverted Dirichlet which has a very restrictive covariance structure. The learning of the parameters of the resulting model is based on the minimization of a message length objective incorporating prior knowledge. We use synthetic data and real data generated from challenging applications, namely visual scenes and objects clustering, to demonstrate the feasibility and advantages of the proposed method.

Bayesian learning of finite generalized inverted Dirichlet mixtures: Application to object classification and forgery detection

The advent of mixture models has opened the possibility of flexible models which are practical to work with. A common assumption is that practitioners typically expect that data are generated from a Gaussian mixture. The inverted Dirichlet mixture has been shown to be a better alternative to the Gaussian mixture and to be of significant value in a variety of applications involving positive data. The inverted Dirichlet is, however, usually undesirable, since it forces an assumption of positive correlation. Our focus here is to develop a Bayesian alternative to both the Gaussian and the inverted Dirichlet mixtures when dealing with positive data. The alternative that we propose is based on the generalized inverted Dirichlet distribution which offers high flexibility and ease of use, as we show in this paper. Moreover, it has a more general covariance structure than the inverted Dirichlet. The proposed mixture model is subjected to a fully Bayesian analysis based on Markov Chain Monte Carlo (MCMC) simulation methods namely Gibbs sampling and Metropolis–Hastings used to compute the posterior distribution of the parameters, and on Bayesian information criterion (BIC) used for model selection. The adoption of this purely Bayesian learning choice is motivated by the fact that Bayesian inference allows to deal with uncertainty in a unified and consistent manner. We evaluate our approach on the basis of two challenging applications concerning object classification and forgery detection.

Bayesian learning of inverted Dirichlet mixtures for SVM kernels generation

We describe approaches for positive data modeling and classification using both finite inverted Dirichlet mixture models and support vector machines (SVMs). Inverted Dirichlet mixture models are used to tackle an outstanding challenge in SVMs namely the generation of accurate kernels. The kernels generation approaches, grounded on ideas from information theory that we consider, allow the incorporation of data structure and its structural constraints. Inverted Dirichlet mixture models are learned within a principled Bayesian framework using both Gibbs sampler and Metropolis-Hastings for parameter estimation and Bayes factor for model selection (i.e., determining the number of mixture’s components). Our Bayesian learning approach uses priors, which we derive by showing that the inverted Dirichlet distribution belongs to the family of exponential distributions, over the model parameters, and then combines these priors with information from the data to build posterior distributions. We illustrate the merits and the effectiveness of the proposed method with two real-world challenging applications namely object detection and visual scenes analysis and classification.

Positive vectors clustering using inverted Dirichlet finite mixture models

In this work we present an unsupervised algorithm for learning finite mixture models from multivariate positive data. Indeed, this kind of data appears naturally in many applications, yet it has not been adequately addressed in the past. This mixture model is based on the inverted Dirichlet distribution, which offers a good representation and modeling of positive non-Gaussian data. The proposed approach for estimating the parameters of an inverted Dirichlet mixture is based on the maximum likelihood (ML) using Newton Raphson method. We also develop an approach, based on the minimum message length (MML) criterion, to select the optimal number of clusters to represent the data using such a mixture. Experimental results are presented using artificial histograms and real data sets. The challenging problem of software modules classification is investigated within the proposed statistical framework, also.