Generalized Dirichlet Distribution Research Articles

Parallel inference for cross-collection latent generalized Dirichlet allocation model and applications

Existing cross-collection topic models with document-topic representation encounter performance bottlenecks in large-scale datasets due to their reliance on Dirichlet priors and conventional inference schemes. These constraints become noticeable in models derived from the Latent Dirichlet Allocation (LDA) framework. To address these challenges, this paper introduces the GPU-accelerated cross-collection latent generalized Dirichlet allocation (gccLGDA) model. This innovative approach integrates the benefits of generalized Dirichlet (GD) distribution with the computational prowess of GPU-based parallel inference, offering enhanced cross-collection topic modeling. The gccLGDA employs the GD distribution presenting a more flexible prior with a comprehensive covariance structure, enabling a more nuanced capture of relationships between latent topics across different collections. Leveraging GPU for parallel inference, our model promises scalable and efficient training for expansive datasets, making it apt for large-scale data challenges. Through empirical evaluations in comparative text mining and document classification, we demonstrate the enhanced performance of the gccLGDA, highlighting its advantages over existing cross-collection topic models.

Online short text clustering using infinite extensions of discrete mixture models

AbstractShort text clustering is one of the fundamental tasks in natural language processing. Different from traditional documents, short texts are ambiguous and sparse due to their short form and the lack of recurrence in word usage from one text to another, making it very challenging to apply conventional machine learning algorithms directly. In this article, we propose two novel approaches for short texts clustering: collapsed Gibbs sampling infinite generalized Dirichlet multinomial mixture model infinite GSGDMM) and collapsed Gibbs sampling infinite Beta‐Liouville multinomial mixture model (infinite GSBLMM). We adopt two flexible and practical priors to the multinomial distribution where in the first one the generalized Dirichlet distribution is integrated, while the second one is based on the Beta‐Liouville distribution. We evaluate the proposed approaches on two famous benchmark datasets, namely, Google News and Tweet. The experimental results demonstrate the effectiveness of our models compared to basic approaches that use Dirichlet priors. We further propose to improve the performance of our methods with an online clustering procedure. We also evaluate the performance of our methods for the outlier detection task, in which we achieve accurate results.

Generalized Dirichlet Distribution Based on Confluent Hypergeometric Series

Dirichlet distribution is a kind of high-dimensional continuous probability distribution, which has important applications in the fields of statistics, machine learning and bioinformatics. In this paper, based on gamma distribution we study two two-dimensional random variables. Then we derive the properties of these two two-dimensional random variables by using the properties of non-central gamma distribution and confluent hypergeometric series. From these properties, we find the two random variables follow generalized Dirichlet distributions. Applying hypergeometric series to Dirichlet distribution broadens the research of Dirichlet distribution.

Decay Branch Ratio Sampling Method with Dirichlet Distribution

The decay branch ratio is evaluated nuclear data related to the decay heat calculation in reactor safety analysis. Decay branch ratio data are inherently subjected to the “sum-to-one” constraint, making it difficult to generate perturbed samples while preserving their suggested statistics in a library of evaluated nuclear data. Therefore, a stochastic-sampling-based uncertainty analysis method is hindered in quantifying the uncertainty contribution of the decay branch ratio to the decay heat calculation. In the present work, two alternative sampling methods are introduced, based on Dirichlet and generalized Dirichlet distribution, to tackle the decay branch ratio sampling issue. The performance of the introduced methods is justified by three-branch decay data retrieved from ENDF/B-VIII.0. The results show that the introduced sampling methods are capable of generating branch ratio samples and preserving their suggested statistics in an evaluated nuclear data library while satisfying their inherent “sum-to-one” constraint. These decay-branch-ratio sampling methods are expected to be alternative procedures in conducting stochastic-sampling-based uncertainty analyses of the decay branch ratio in reactor simulations.

Smoothed Generalized Dirichlet: A Novel Count-Data Model for Detecting Emotional States

In this article, we propose novel approaches to deal with the problem of burstiness, the challenge of count-data sparseness, and the curse of dimensionality. We introduce a smoothed generalized Dirichlet distribution that is a smoothed variant of the generalized Dirichlet distribution and a generalization of the smoothed Dirichlet. We provide different learning methods based on mixture models and agglomerative clustering-based geometrical information: Kullback–Leibler divergence, Fisher metric, and Bhattacharyya distance. Moreover, we show that the new smoothed generalized Dirichlet could be considered as a prior to the multinomial, which generates a new distribution for count data that we call the smoothed generalized Dirichlet multinomial. In particular, we present an approximation based on Taylor series expansion for better performance and optimized running time in the case of high-dimensional count data. The proposed models are evaluated through two emotion detection applications: disaster-tweet-related emotions and pain intensity estimation. Experiments show the efficiency and the robustness of our approaches when dealing with texts, videos, and images.

Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

The high accuracy attainment, using less complex architectures of neural networks, remains one of the most important problems in machine learning. In many studies, increasing the quality of recognition and prediction is obtained by extending neural networks with usual or special neurons, which significantly increases the time of training. However, engaging an optimization algorithm, which gives us a value of the loss function in the neighborhood of global minimum, can reduce the number of layers and epochs. In this work, we explore the extreme searching of multidimensional functions by proposed natural gradient descent based on Dirichlet and generalized Dirichlet distributions. The natural gradient is based on describing a multidimensional surface with probability distributions, which allows us to reduce the change in the accuracy of gradient and step size. The proposed algorithm is equipped with step-size adaptation, which allows it to obtain higher accuracy, taking a small number of iterations in the process of minimization, compared with the usual gradient descent and adaptive moment estimate. We provide experiments on test functions in four- and three-dimensional spaces, where natural gradient descent proves its ability to converge in the neighborhood of global minimum. Such an approach can find its application in minimizing the loss function in various types of neural networks, such as convolution, recurrent, spiking and quantum networks.

Multinomial naïve Bayesian classifier with generalized Dirichlet priors for high-dimensional imbalanced data

The data-level approach can be used to balance the class distribution in a data set, while generating synthetic instances in a high-dimensional space is an awful task. Naïve Bayesian classifier with multinomial model, called multinomial naïve Bayesian classifier, is a popular classification algorithm for high-dimensional data because of its computational efficiency. However, this algorithm generally cannot have a better performance than the other algorithms on high-dimensional imbalanced data. Generalized Dirichlet distribution that is defined on unit simplex can be priors for multinomial naïve Bayesian classifier. This study proposes methods to find noninformative generalized Dirichlet priors for multinomial naïve Bayesian classifier so that its performance on high-dimensional imbalanced data can be largely improved. The methods are tested on seven high-dimensional imbalanced data sets to demonstrate that the multinomial naïve Bayesian classifier with generalized Dirichlet priors can significantly outperform not only the original multinomial naïve Bayesian classifier, but also random forest and Ripper algorithm.

A Fractional Generalization of the Dirichlet Distribution and Related Distributions

Abstract This paper is devoted to a fractional generalization of the Dirichlet distribution. The form of the multivariate distribution is derived assuming that the n partitions of the interval [0, Wn ] are independent and identically distributed random variables following the generalized Mittag-Leffler distribution. The expected value and variance of the one-dimensional marginal are derived as well as the form of its probability density function. A related generalized Dirichlet distribution is studied that provides a reasonable approximation for some values of the parameters. The relation between this distribution and other generalizations of the Dirichlet distribution is discussed. Monte Carlo simulations of the one-dimensional marginals for both distributions are presented.

Stochastic topic models for large scale and nonstationary data

Many traditional database’s processing schemes are batch-based with their abilities to utilize the entire information available at a time. Though, their limitations include storage (memory issues) and computational speed (often slow) for large scale applications. Another major disadvantage of the batch processing is that any small change or update in the database often requires a reevaluation using all the data at a time. This is not efficient as it is time consuming and exhausting. So, the approach seems to be a little obsolete in this new generation of fast computation. Furthermore and recently, the decrease in the cost of performing computations online promoted the increase in streaming and online-based models. In other words, new systems are taking advantage of the online setting to build models that are able to perform in real time and handle fast computations with real time updates. Traditional models could no longer scale to very large applications. So, much support has been given to online framework as these massive and nonstationary data could not keep up with the available storage. In the case of generative models, usually, the lack of flexible priors and sometimes the high complexities in the methods often hindered their performances. In addition and most importantly, many online-based models still use traditional inference approaches such as variational Bayes (VB) and Markov chain Monte Carlo (MCMC) which individually are not flexible enough as they suffer from either accuracy or efficiency. As a result, we propose in this paper, a new model that operates in online fashion with BL (Beta-Liouville) prior due to its flexibilities in topic correlation analysis. Carrying only very few parameters (compared to the generalized Dirichlet distribution, for instance), the BL is now coupled with a robust and stochastic generative process within a new hybrid inference that combines only the advantages of the VB and Gibbs sampling in the collapsed space. This insures an efficient, fast, and accurate processing. Experimental results with nonstationary datasets for face detection, image classification, and text documents processing show the merits of the new stochastic approach.

Variational-based latent generalized Dirichlet allocation model in the collapsed space and applications

In topic modeling framework, many Dirichlet-based models performances have been hindered by the limitations of the conjugate prior. It led to models with more flexible priors, such as the generalized Dirichlet distribution, that tend to capture semantic relationships between topics (topic correlation). Now these extensions also suffer from incomplete generative processes that complicate performances in traditional inferences such as VB (Variational Bayes) and CGS (Collaspsed Gibbs Sampling). As a result, the new approach, the CVB-LGDA (Collapsed Variational Bayesian inference for the Latent Generalized Dirichlet Allocation) presents a scheme that integrates a complete generative process to a robust inference technique for topic correlation and codebook analysis. Its performance in image classification, facial expression recognition, 3D objects categorization, and action recognition in videos shows its merits.

Data-free metrics for Dirichlet and generalized Dirichlet mixture-based HMMs – A practical study

Approaches to design metrics between hidden Markov models (HMM) can be divided into two classes: data-based and parameter-based. The latter has the clear advantage of being deterministic and faster but only a very few similarity measures that can be applied to mixture-based HMMs have been proposed so far. Most of these metrics apply to the discrete or Gaussian HMMs and no comparative study have been led to the best of our knowledge. With the recent development of HMMs based on the Dirichlet and generalized Dirichlet distributions for proportional data modeling, we propose to design three new parametric similarity measures between these HMMs. Extensive experiments on synthetic data show the reliability of these new measures where the existing ones fail at giving expected results when some parameters vary. Illustration on real data show the clustering capability of these measures and their potential applications.

Small Area Estimation of Proportions with Constraint for National Resources Inventory Survey

Motivated by the need to produce small area estimates for the National Resources Inventory survey, we develop a spatial hierarchical model based on the generalized Dirichlet distribution to construct small area estimators of compositional proportions in several mutually exclusive and exhaustive landcover categories. At the observation level, the standard design-based estimators of the proportions are assumed to follow the generalized Dirichlet distribution. After proper transformation of the design-based estimators, beta regression is applicable. We consider a logit mixed model for the expectation of the beta distribution, which incorporates covariates through fixed effects and spatial effect through a conditionally autoregressive process. In a design-based evaluation study, the proposed model-based estimators are shown to have smaller root-mean-square error and relative root-mean-square error than design-based estimators and multinomial model-based estimators. Supplementary materials accompanying this paper appear online.

Proportional data modeling via entropy-based variational bayes learning of mixture models

During the last few decades, many statistical approaches that were developed in the fields of computer vision and pattern recognition are based on mixture models. A mixture-based representation has a number of advantages: mixture models are generative, flexible, plus they can take prior information into account to improve the generalization capability. The mixture models that we consider in this paper are based on the Dirichlet and generalized Dirichlet distributions that have been widely used to represent proportional data. The novel aspect of this paper is to develop an entropy-based framework to learn these mixture models. Specifically, we propose a Bayesian framework for model learning by means of a sophisticated entropy-based variational Bayes technique. We present experimental results to show that the proposed method is effective in several applications namely person identity verification, 3D object recognition, text document clustering, and gene expression categorization.

Online Learning of Hierarchical Pitman-Yor Process Mixture of Generalized Dirichlet Distributions With Feature Selection.

In this paper, a novel statistical generative model based on hierarchical Pitman-Yor process and generalized Dirichlet distributions (GDs) is presented. The proposed model allows us to perform joint clustering and feature selection thanks to the interesting properties of the GD distribution. We develop an online variational inference algorithm, formulated in terms of the minimization of a Kullback-Leibler divergence, of our resulting model that tackles the problem of learning from high-dimensional examples. This variational Bayes formulation allows simultaneously estimating the parameters, determining the model's complexity, and selecting the appropriate relevant features for the clustering structure. Moreover, the proposed online learning algorithm allows data instances to be processed in a sequential manner, which is critical for large-scale and real-time applications. Experiments conducted using challenging applications, namely, scene recognition and video segmentation, where our approach is viewed as an unsupervised technique for visual learning in high-dimensional spaces, showed that the proposed approach is suitable and promising.

Expectation propagation learning of a Dirichlet process mixture of Beta-Liouville distributions for proportional data clustering

We propose a nonparametric Bayesian model for the clustering of proportional data. Our model is based on an infinite mixture of Beta-Liouville distributions and allows a compact description of complex data. The choice of the Beta-Liouville as the basis of our model is justified by the fact that it has been shown to be a good alternative to the Dirichlet and generalized Dirichlet distributions for the statistical representation of proportional data. Using this infinite mixture, we show how a careful modeling can achieve good results by allowing the elicitation of prior belief about the parameters and the number of clusters through suitable learning. Indeed, we develop an efficient learning algorithm, based on expectation propagation, to estimate the parameters of our infinite Beta-Liouville mixture model. The feasibility and effectiveness of the proposed method are demonstrated by two challenging applications namely action and facial expression recognition.

A hierarchical Dirichlet process mixture of generalized Dirichlet distributions for feature selection

This paper addresses the problem of identifying meaningful patterns and trends in data via clustering (i.e. automatically dividing a data set into meaningful homogenous sub-groups such that the data within the same sub-group are very similar, and data in different sub-groups are very different). The clustering framework that we propose is based on the generalized Dirichlet distribution, which is widely accepted as a flexible modeling approach, and a hierarchical Dirichlet process mixture prior. A main advantage of the adopted hierarchical Dirichlet process is that it provides a principled elegant nonparametric Bayesian approach to model selection by supposing that the number of mixture components can go to infinity. In addition to capturing the structure of the data, the combination of hierarchical Dirichlet process and generalized Dirichlet distribution is shown to offer a natural efficient solution to the feature selection problem when dealing with high-dimensional data. We develop two variational learning approaches (i.e. batch and incremental) for learning the parameters of the proposed model. The batch algorithm examines the entire data set at once while the incremental one learns the model one step at a time (i.e. update the model’s parameters each time new data are introduced). The utility of the proposed approach is demonstrated on real applications namely face detection, facial expression recognition, human gesture recognition, and off-line writer identification. The obtained results show clearly the merits of our statistical framework.

Variational learning of hierarchical infinite generalized Dirichlet mixture models and applications

Data clustering is a fundamental unsupervised learning task in several domains such as data mining, computer vision, information retrieval, and pattern recognition. In this paper, we propose and analyze a new clustering approach based on both hierarchical Dirichlet processes and the generalized Dirichlet distribution, which leads to an interesting statistical framework for data analysis and modelling. Our approach can be viewed as a hierarchical extension of the infinite generalized Dirichlet mixture model previously proposed in Bouguila and Ziou (IEEE Trans Neural Netw 21(1):107---122, 2010). The proposed clustering approach tackles the problem of modelling grouped data where observations are organized into groups that we allow to remain statistically linked by sharing mixture components. The resulting clustering model is learned using a principled variational Bayes inference-based algorithm that we have developed. Extensive experiments and simulations, based on two challenging applications namely images categorization and web service intrusion detection, demonstrate our model usefulness and merits.

A variational Bayes model for count data learning and classification

Several machine learning and knowledge discovery approaches have been proposed for count data modeling and classification. In particular, latent Dirichlet allocation (LDA) (Blei et al., 2003a) has received a lot of attention and has been shown to be extremely useful in several applications. Although the LDA is generally accepted to be one of the most powerful generative models, it is based on the Dirichlet assumption which has some drawbacks as we shall see in this paper. Thus, our goal is to enhance the LDA by considering the generalized Dirichlet distribution as a prior. The resulting generative model is named latent generalized Dirichlet allocation (LGDA) to maintain consistency with the original model. The LGDA is learned using variational Bayes which provides computationally tractable posterior distributions over the model׳s hidden variables and its parameters. To evaluate the practicality and merits of our approach, we consider two challenging applications namely text classification and visual scene categorization.

Bayesian nowcasting during the STEC O104:H4 outbreak in Germany, 2011.

A Bayesian approach to the prediction of occurred-but-not-yet-reported events is developed for application in real-time public health surveillance. The motivation was the prediction of the daily number of hospitalizations for the hemolytic-uremic syndrome during the large May-July 2011 outbreak of Shiga toxin-producing Escherichia coli (STEC) O104:H4 in Germany. Our novel Bayesian approach addresses the count data nature of the problem using negative binomial sampling and shows that right-truncation of the reporting delay distribution under an assumption of time-homogeneity can be handled in a conjugate prior-posterior framework using the generalized Dirichlet distribution. Since, in retrospect, the true number of hospitalizations is available, proper scoring rules for count data are used to evaluate and compare the predictive quality of the procedures during the outbreak. The results show that it is important to take the count nature of the time series into account and that changes in the delay distribution occurred due to intervention measures. As a consequence, we extend the Bayesian analysis to a hierarchical model, which combines a discrete time survival regression model for the delay distribution with a penalized spline for the dynamics of the epidemic curve. Altogether, we conclude that in emerging and time-critical outbreaks, nowcasting approaches are a valuable tool to gain information about current trends.

Unsupervised clustering and feature weighting based on Generalized Dirichlet mixture modeling

We propose a possibilistic approach for Generalized Dirichlet mixture parameter estimation, data clustering, and feature weighting. The proposed algorithm, called Robust and Unsupervised Learning of Finite Generalized Dirichlet Mixture Models (RULe_GDM), exploits a property of the Generalized Dirichlet distributions that transforms the data to make the features independent and follow Beta distributions. Then, it learns optimal relevance weights for each feature within each cluster. This property makes RULe_GDM suitable for noisy and high-dimensional feature spaces. In addition, RULe_GDM associates two types of memberships with each data sample. The first one is the posterior probability and indicates how well a sample fits each estimated distribution. The second membership represents the degree of typicality and is used to identify and discard noise points and outliers. RULe_GDM minimizes one objective function which combines learning the two membership functions, distribution parameters, and the relevance weights for each feature within each distribution. We also extend our algorithm to find the optimal number of clusters in an unsupervised and efficient way by exploiting some properties of the possibilistic membership function. The performance of RULe_GDM is illustrated and compared to similar algorithms. We use synthetic data to illustrate its robustness to noisy and high dimensional features. We also compare our approach to other relevant algorithms using several standard data sets.