Inverted Dirichlet Research Articles

Characterization of the Three-Variate Inverted Dirichlet Distributions

In this paper, we prove a characterization of three-variate inverted Dirichlet distributions by an independence property. The main technical challenge was a problem involving the solution of a related functional equation.

Dirichlet Process Mixture of Generalized Inverted Dirichlet Distributions for Positive Vector Data With Extended Variational Inference.

A Bayesian nonparametric approach for estimation of a Dirichlet process (DP) mixture of generalized inverted Dirichlet distributions [i.e., an infinite generalized inverted Dirichlet mixture model (InGIDMM)] has been proposed. The generalized inverted Dirichlet distribution has been proven to be efficient in modeling the vectors that contain only positive elements. Under the classical variational inference (VI) framework, the key challenge in the Bayesian estimation of InGIDMM is that the expectation of the joint distribution of data and variables cannot be explicitly calculated. Therefore, numerical methods are usually applied to simulate the optimal posterior distributions. With the recently proposed extended VI (EVI) framework, we introduce lower bound approximations to the original variational objective function in the VI framework such that an analytically tractable solution can be derived. Hence, the problem in numerical simulation has been overcome. By applying the DP mixture technique, an InGIDMM can automatically determine the number of mixture components from the observed data. Moreover, the DP mixture model with an infinite number of mixture components also avoids the problems of underfitting and overfitting. The performance of the proposed approach is demonstrated with both synthesized data and real-life data applications.

Simultaneous positive sequential vectors modeling and unsupervised feature selection via continuous hidden Markov models

Since positive data vectors are often naturally generated in various real-life applications, positive vectors modeling has become an important research topic. In this article, we tackle the problem of modeling positive sequential vectors through continuous hidden Markov models (HMMs). Motivated by several recent studies in which the generalized inverted Dirichlet (GID) distribution has provided better performance than the Gaussian distribution for modeling positive data, instead of adopting Gaussian mixture models (GMM) as the emission density for conventional continuous HMMs, we theoretically propose a novel HMM by considering the mixture of GID distributions as the emission density. Moreover, to cope with high-dimensional data which may contain irrelevant features, an unsupervised localized feature selection method is incorporated with our model, which results in a unified framework that can simultaneously perform positive sequential data modeling and feature selection. To learn the proposed model, we develop a convergence-guaranteed algorithm based on variational Bayes. The advantages of our model are demonstrated through both simulated data sets and a real-life application about human action recognition.

Unsupervised Learning Using Variational Inference on Finite Inverted Dirichlet Mixture Models with Component Splitting

Unsupervised learning has been one of the essentials of pattern recognition and data mining. The role of Dirichlet family of mixture models in this field is inevitable. In this article, we propose a finite Inverted Dirichlet mixture model for unsupervised learning using variational inference. In particular, we develop an incremental algorithm with a component splitting approach for local model selection, which makes the clustering algorithm more efficient. We illustrate our model and learning algorithm with synthetic data and some real applications for occupancy estimation in smart homes and topic learning in images and videos. Extensive comparisons with comparable recent approaches have shown the merits of our proposed model.

A novel approach for modeling positive vectors with inverted Dirichlet-based hidden Markov models

Hidden Markov Models (HMMs) are among the most remarkably powerful probabilistic models, that although been acknowledged for decades have recently made a huge resurgence in the machine learning field. Their ever-growing use to model diversified and heterogeneous data (image,video, audio, time series) in numerous important practical situations is the subject of all forms of perpetual extensions. This work presents what we believe to be the first integration of the Inverted Dirichlet (ID) Mixture Models into the framework of HMMs. The proposed method uses the inverted Dirichlet mixtures to model the emission probabilities also known as observation probabilities. This extension (IDHMM), is motivated by the proven capacity of these mixtures to deal with positive vectors and overcome mixture models’ capability to take into account any ordering or temporal constraints relative to information. The complete inference and parameter estimation are detailed in this work. Applications in the context of image categorization and indoor occupancy detection demonstrate higher performance compared to the extensively used Gaussian mixture-based Hidden Markov Model (GHMM) and the Dirichlet mixture-based hidden Markov Model (DHMM).

Nonparametric Hierarchical Bayesian Models for Positive Data Clustering Based on Inverted Dirichlet-Based Distributions

In this paper, we propose nonparametric hierarchical Bayesian models based on two inverted Dirichlet-based distributions and Pitman-Yor process for positive data features clustering. The choice of the inverted Dirichlet and the generalized inverted Dirichlet distributions is motivated by their flexibility and modeling capabilities when dealing with this kind of data, while deploying the Pitman-Yor process prior is justified by its power-law behavior, which makes it a natural choice in real-life application compared with Dirichlet processes for instance. The inference for the resulting models takes into account the challenging problem of feature weighting/selection and is conducted under a Bayesian setting by means of the recently proposed stochastic variational Bayes technique. The efficacy and merits of the proposed approaches are examined using the synthetic data and a challenging real-life application that concerns video background subtraction.

Insights Into Multiple/Single Lower Bound Approximation for Extended Variational Inference in Non-Gaussian Structured Data Modeling.

For most of the non-Gaussian statistical models, the data being modeled represent strongly structured properties, such as scalar data with bounded support (e.g., beta distribution), vector data with unit length (e.g., Dirichlet distribution), and vector data with positive elements (e.g., generalized inverted Dirichlet distribution). In practical implementations of non-Gaussian statistical models, it is infeasible to find an analytically tractable solution to estimating the posterior distributions of the parameters. Variational inference (VI) is a widely used framework in Bayesian estimation. Recently, an improved framework, namely, the extended VI (EVI), has been introduced and applied successfully to a number of non-Gaussian statistical models. EVI derives analytically tractable solutions by introducing lower bound approximations to the variational objective function. In this paper, we compare two approximation strategies, namely, the multiple lower bounds (MLBs) approximation and the single lower bound (SLB) approximation, which can be applied to carry out the EVI. For implementation, two different conditions, the weak and the strong conditions, are discussed. Convergence of the EVI depends on the selection of the lower bound, regardless of the choice of weak or strong condition. We also discuss the convergence properties to clarify the differences between MLB and SLB. Extensive comparisons are made based on some EVI-based non-Gaussian statistical models. Theoretical analysis is conducted to demonstrate the differences between the weak and strong conditions. Experimental results based on real data show advantages of the SLB approximation over the MLB approximation.

Variational Bayesian Learning for Dirichlet Process Mixture of Inverted Dirichlet Distributions in Non-Gaussian Image Feature Modeling.

In this paper, we develop a novel variational Bayesian learning method for the Dirichlet process (DP) mixture of the inverted Dirichlet distributions, which has been shown to be very flexible for modeling vectors with positive elements. The recently proposed extended variational inference (EVI) framework is adopted to derive an analytically tractable solution. The convergency of the proposed algorithm is theoretically guaranteed by introducing single lower bound approximation to the original objective function in the EVI framework. In principle, the proposed model can be viewed as an infinite inverted Dirichlet mixture model that allows the automatic determination of the number of mixture components from data. Therefore, the problem of predetermining the optimal number of mixing components has been overcome. Moreover, the problems of overfitting and underfitting are avoided by the Bayesian estimation approach. Compared with several recently proposed DP-related methods and conventional applied methods, the good performance and effectiveness of the proposed method have been demonstrated with both synthesized data and real data evaluations.

Bayesian inference by reversible jump MCMC for clustering based on finite generalized inverted Dirichlet mixtures

The goal of constructing models from examples has been approached from different perspectives. Statistical methods have been widely used and proved effective in generating accurate models. Finite Gaussian mixture models have been widely used to describe a wide variety of random phenomena and have played a prominent role in many attempts to develop expressive statistical models in machine learning. However, their effectiveness is limited to applications where underlying modeling assumptions (e.g., the per-components densities are Gaussian) are reasonably satisfied. Thus, much research efforts have been devoted to developing better alternatives. In this paper, we focus on constructing statistical models from positive vectors (i.e., vectors whose elements are strictly greater than zero) for which the generalized inverted Dirichlet (GID) mixture has been shown to be a flexible and powerful parametric framework. In particular, we propose a Bayesian density estimation method based upon mixtures of GIDs. The consideration of Bayesian learning is interesting in several respects. It allows to take uncertainty into account by introducing prior information about the parameters, it allows simultaneous parameters estimation and model selection, and it allows to overcome learning problems related to over- or under-fitting. Indeed, we develop a reversible jump Markov Chain Monte Carlo sampler for GID mixtures that we apply for simultaneous clustering and feature selection in the context of some challenging real-world applications concerning scene classification, action recognition, and video forgery detection.

Variational Bayesian Inference for Finite Inverted Dirichlet Mixture Model and Its Application to Object Detection

As a variant of Finite mixture model (FMM), finite Inverted Dirichlet mixture model (IDMM) can not avoid the conventional challenges, such as how to select the appropriate number of mixture components based on the observed data. Towards easing these issues, we propose a variational inference framework for learning IDMM which has been proved to be an efficient tool for modeling vectors with positive elements. Compared with the conventional Expectation maximization (EM) algorithm commonly used for learning FMM, the proposed approach prevents over-fitting well. Furthermore, it is able to do automatic determination of the number of mixture components and parameters estimation, simultaneously. Experimental results on both synthetic and real data of object detection confirm significant improvements on flexibility and efficiency being achieved.

A statistical framework for online learning using adjustable model selection criteria

Model-based approaches have been for long an effective method to model data and classify it. Recently they have been used to model users interactions with a given system in order to satisfy their needs through adequate responses. The semantic gap between the system and the user perception for the data makes this modeling hard to be designed based on the features space only. Indeed the user intervention is somehow needed to inform the system how the data should be perceived according to some ontology and hierarchy when new data are introduced to the model. Such a task is challenging as the system should learn how to establish the update according to the user perception and representation of the data. In this work, we propose a new methodology to update a mixture model based on the generalized inverted Dirichlet distribution, that takes into account simultaneously user׳s perception and the dynamic nature of real-world data. Experiments on synthetic data as well as real data generated from a challenging application namely visual objects classification indicate that the proposed approach has merits and provides promising results.

Variational Bayesian inference for infinite generalized inverted Dirichlet mixtures with feature selection and its application to clustering

We developed a variational Bayesian learning framework for the infinite generalized Dirichlet mixture model (i.e. a weighted mixture of Dirichlet process priors based on the generalized inverted Di...

Robust simultaneous positive data clustering and unsupervised feature selection using generalized inverted Dirichlet mixture models

The discovery, extraction and analysis of knowledge from data rely generally upon the use of unsupervised learning methods, in particular clustering approaches. Much recent research in clustering and data engineering has focused on the consideration of finite mixture models which allow to reason in the face of uncertainty and to learn by example. The adoption of these models becomes a challenging task in the presence of outliers and in the case of high-dimensional data which necessitates the deployment of feature selection techniques. In this paper we tackle simultaneously the problems of cluster validation (i.e. model selection), feature selection and outliers rejection when clustering positive data. The proposed statistical framework is based on the generalized inverted Dirichlet distribution that offers a more practical and flexible alternative to the inverted Dirichlet which has a very restrictive covariance structure. The learning of the parameters of the resulting model is based on the minimization of a message length objective incorporating prior knowledge. We use synthetic data and real data generated from challenging applications, namely visual scenes and objects clustering, to demonstrate the feasibility and advantages of the proposed method.

Bayesian learning of finite generalized inverted Dirichlet mixtures: Application to object classification and forgery detection

The advent of mixture models has opened the possibility of flexible models which are practical to work with. A common assumption is that practitioners typically expect that data are generated from a Gaussian mixture. The inverted Dirichlet mixture has been shown to be a better alternative to the Gaussian mixture and to be of significant value in a variety of applications involving positive data. The inverted Dirichlet is, however, usually undesirable, since it forces an assumption of positive correlation. Our focus here is to develop a Bayesian alternative to both the Gaussian and the inverted Dirichlet mixtures when dealing with positive data. The alternative that we propose is based on the generalized inverted Dirichlet distribution which offers high flexibility and ease of use, as we show in this paper. Moreover, it has a more general covariance structure than the inverted Dirichlet. The proposed mixture model is subjected to a fully Bayesian analysis based on Markov Chain Monte Carlo (MCMC) simulation methods namely Gibbs sampling and Metropolis–Hastings used to compute the posterior distribution of the parameters, and on Bayesian information criterion (BIC) used for model selection. The adoption of this purely Bayesian learning choice is motivated by the fact that Bayesian inference allows to deal with uncertainty in a unified and consistent manner. We evaluate our approach on the basis of two challenging applications concerning object classification and forgery detection.

Object clustering and recognition using multi-finite mixtures for semantic classes and hierarchy modeling

Model-based approaches have become important tools to model data and infer knowledge. Such approaches are often used for clustering and object recognition which are crucial steps in many applications, including but not limited to, recommendation systems, search engines, cyber security, surveillance and object tracking. Many of these applications have the urgent need to reduce the semantic gap of data representation between the system level and the human being understandable level. Indeed, the low level features extracted to represent a given object can be confusing to machines which cannot differentiate between very similar objects trivially distinguishable by human beings (e.g. apple vs. tomato). In this paper, we propose a novel hierarchical methodology for data representation using a hierarchical mixture model. The proposed approach allows to model a given object class by a set of modes deduced by the system and grouped according to a labeled training data representing the human level semantic. We have used the inverted Dirichlet distribution to build our statistical framework. The proposed approach has been validated using both synthetic data and a challenging application namely visual object clustering and recognition. The presented model is shown to have a flexible hierarchy that can be changed on the fly within costless computational time.

Bayesian learning of inverted Dirichlet mixtures for SVM kernels generation

We describe approaches for positive data modeling and classification using both finite inverted Dirichlet mixture models and support vector machines (SVMs). Inverted Dirichlet mixture models are used to tackle an outstanding challenge in SVMs namely the generation of accurate kernels. The kernels generation approaches, grounded on ideas from information theory that we consider, allow the incorporation of data structure and its structural constraints. Inverted Dirichlet mixture models are learned within a principled Bayesian framework using both Gibbs sampler and Metropolis-Hastings for parameter estimation and Bayes factor for model selection (i.e., determining the number of mixture’s components). Our Bayesian learning approach uses priors, which we derive by showing that the inverted Dirichlet distribution belongs to the family of exponential distributions, over the model parameters, and then combines these priors with information from the data to build posterior distributions. We illustrate the merits and the effectiveness of the proposed method with two real-world challenging applications namely object detection and visual scenes analysis and classification.

Positive vectors clustering using inverted Dirichlet finite mixture models

In this work we present an unsupervised algorithm for learning finite mixture models from multivariate positive data. Indeed, this kind of data appears naturally in many applications, yet it has not been adequately addressed in the past. This mixture model is based on the inverted Dirichlet distribution, which offers a good representation and modeling of positive non-Gaussian data. The proposed approach for estimating the parameters of an inverted Dirichlet mixture is based on the maximum likelihood (ML) using Newton Raphson method. We also develop an approach, based on the minimum message length (MML) criterion, to select the optimal number of clusters to represent the data using such a mixture. Experimental results are presented using artificial histograms and real data sets. The challenging problem of software modules classification is investigated within the proposed statistical framework, also.

A TRUNCATED BIVARIATE INVERTED DIRICHLET DISTRIBUTION

A truncated version of the bivariate inverted dirichlet distribution is introduced. Unlike the inverted dirichlet distribution, this possesses finite moments of all orders and could therefore be a better model for certain practical situations. One such situation is discussed. The moments, maximum likelihood estimators and the Fisher information matrix for the truncated distribution are derived.

Some Truncated Bivariate Distributions

Truncated versions of the bivariate generalized Pareto, bivariate inverted dirichlet and the bivariate Pearson type VII distributions are introduced. Unlike the un-truncated versions, these possess finite moments of all orders and could therefore be better models for certain practical situations. Explicit expressions for the moments are derived for each of the truncated distribution.

Exact and Approximate Distributions for the Linear Combination of Inverted Dirichlet Components

It is well known that X + Y has the F distribution when X and Y follow the inverted Dirichlet distribution. In this paper, we derive the exact distribution of the general form αX + βY (involving the Gauss hypergeometric function)and the corresponding moment properties. We also propose approximations and discuss evidence of their robustness based on the powerful Kolmogorov-Smirnov test. The work is motivated by real-life examples in quality and reliability engineering.