Hierarchical Pitman-Yor Process Research Articles

Joint modeling of causal phrases-sentiments-aspects using Hierarchical Pitman Yor Process

Traditional sentiment-aware topic models assume that topic or sentiment transition occurs from either a sentence to the next sentence or from a word to the next word. Such models cannot capture a topic or sentiment transition at phrase boundaries. Further, most of the models adopt a sentiment lexicon to initialize sentiment priors and this approach induces coverage problems. To overcome the above-cited limitations, we have proposed a topic model that extracts aspects, sentiments, and causal phrases simultaneously by leveraging Hierarchical Pitman Yor Process (HPYP) that is modified using a sentiment component, a word-tagger to guide the causal phrase generation and a sentiment prior initialized through a sequential model to address coverage problems. We have evaluated our model on six datasets and found that the proposed model outperforms the baselines in terms of perplexity by 14%, topical coherence by 20%, topic diversity by 5%, sentiment classification task’s accuracy by 4% and, precision, recall and F1 score by 2%. Ablation studies assert that sequence model based sentiment prior initialization results in increasing the accuracy of sentiment classification by 2%.

Grouped Spherical Data Modeling Through Hierarchical Nonparametric Bayesian Models and Its Application to fMRI Data Analysis.

Recently, spherical data (i.e., L2 normalized vectors) modeling has become a promising research topic in various real-world applications (such as gene expression data analysis, document categorization, and gesture recognition). In this work, we propose a hierarchical nonparametric Bayesian model based on von Mises-Fisher (VMF) distributions for modeling spherical data that involve multiple groups, where each observation within a group is sampled from a VMF mixture model with an infinite number of components allowing them to be shared across groups. Our model is formulated by employing a hierarchical nonparametric Bayesian framework known as the hierarchical Pitman-Yor (HPY) process mixture model, which possesses a power-law nature over the distribution of the components and is particularly useful for data distributions with heavy tails and skewness. To learn the proposed HPY process mixture model with VMF distributions, we systematically develop a closed-form optimization algorithm based on variational Bayes (VB). The merits of the proposed hierarchical Bayesian nonparametric model for modeling grouped spherical data are demonstrated through experiments on both synthetic data and a real-world application about resting-state functional magnetic resonance imaging (fMRI) data analysis.

Unsupervised Grouped Axial Data Modeling via Hierarchical Bayesian Nonparametric Models With Watson Distributions.

This paper aims at proposing an unsupervised hierarchical nonparametric Bayesian framework for modeling axial data (i.e., observations are axes of direction) that can be partitioned into multiple groups, where each observation within a group is sampled from a mixture of Watson distributions with an infinite number of components that are allowed to be shared across different groups. First, we propose a hierarchical nonparametric Bayesian model for modeling grouped axial data based on the hierarchical Pitman-Yor process mixture model of Watson distributions. Then, we demonstrate that by setting the discount parameters of the proposed model to 0, another hierarchical nonparametric Bayesian model based on hierarchical Dirichlet process can be derived for modeling axial data. To learn the proposed models, we systematically develop a closed-form optimization algorithm based on the collapsed variational Bayes (CVB) inference. Furthermore, to ensure the convergence of the proposed learning algorithm, an annealing mechanism is introduced to the framework of CVB inference, leading to an averaged collapsed variational Bayes inference strategy. The merits of the proposed models for modeling grouped axial data are demonstrated through experiments on both synthetic data and real-world applications involving gene expression data clustering and depth image analysis.

Hierarchical Dirichlet and Pitman–Yor process mixtures of shifted‐scaled Dirichlet distributions for proportional data modeling

AbstractIn this article, first, we propose a novel unsupervised learning method based on a hierarchical Dirichlet process mixture of shifted‐scaled Dirichlet (SSD) distributions. Second, we extend it to a hierarchical Pitman–Yor process mixture of SSD distributions. The goal is to find a model that properly fits complex real‐world data. Our models are based on SSD distributions that are more flexible than Dirichlet distribution in fitting proportional data. Simultaneous data fitting (parameter estimate) and model selection (model complexity determination) are possible with the suggested methods. We applied batch and online variational inference for learning the models. The online setting allows us to feed our models with large‐scale streaming data. The effectiveness of our proposed models is evaluated by four realistic and challenging applications, namely, spam email detection, texture clustering, traffic sign detection, and vehicle detection. Experimental results demonstrate the potential of our models to fit proportional data.

Perfect Sampling of the Posterior in the Hierarchical Pitman-Yor Process.

The predictive probabilities of the hierarchical Pitman-Yor process are approximated through Monte Carlo algorithms that exploits the Chinese Restaurant Franchise (CRF) representation. However, in order to simulate the posterior distribution of the hierarchical Pitman-Yor process, a set of auxiliary variables representing the arrangement of customers in tables of the CRF must be sampled through Markov chain Monte Carlo. This paper develops a perfect sampler for these latent variables employing ideas from the Propp-Wilson algorithm and evaluates its average running time by extensive simulations. The simulations reveal a significant dependence of running time on the parameters of the model, which exhibits sharp transitions. The algorithm is compared to simpler Gibbs sampling procedures, as well as a procedure for unbiased Monte Carlo estimation proposed by Glynn and Rhee. We illustrate its use with an example in microbial genomics studies.

Hierarchical viewpoint discovery from tweets using Bayesian modelling

When users express their stances towards a topic in social media, they might elaborate their viewpoints or reasoning. Oftentimes, viewpoints expressed by different users exhibit a hierarchical structure. Therefore, detecting this kind of hierarchical viewpoints offers a better insight to understand the public opinion. In this paper, we propose a novel Bayesian model for hierarchical viewpoint discovery from tweets. Driven by the motivation that a viewpoint expressed in a tweet can be regarded as a path from the root to a leaf of a hierarchical viewpoint tree, the assignment of the relevant viewpoint topics is assumed to follow two nested Chinese restaurant processes. Moreover, opinions in text are often expressed in un-semantically decomposable multi-terms or phrases, such as ‘economic recession’. Hence, a hierarchical Pitman–Yor process is employed as a prior for modelling the generation of phrases with arbitrary length. Experimental results on two Twitter corpora demonstrate the effectiveness of the proposed Bayesian model for hierarchical viewpoint discovery.

Multi-Armed Bandit for Species Discovery: A Bayesian Nonparametric Approach

ABSTRACTLet (P1, …, PJ) denote J populations of animals from distinct regions. A priori, it is unknown which species are present in each region and what are their corresponding frequencies. Species are shared among populations and each species can be present in more than one region with its frequency varying across populations. In this article, we consider the problem of sequentially sampling these populations to observe the greatest number of different species. We adopt a Bayesian nonparametric approach and endow (P1, …, PJ) with a hierarchical Pitman–Yor process prior. As a consequence of the hierarchical structure, the J unknown discrete probability measures share the same support, that of their common random base measure. Given this prior choice, we propose a sequential rule that, at every time step, given the information available up to that point, selects the population from which to collect the next observation. Rather than picking the population with the highest posterior estimate of producing a new value, the proposed rule includes a Thompson sampling step to better balance the exploration–exploitation trade-off. We also propose an extension of the algorithm to deal with incidence data, where multiple observations are collected in a time period. The performance of the proposed algorithms is assessed through a simulation study and compared to three other strategies. Finally, we compare these algorithms using a dataset of species of trees, collected from different plots in South America. Supplementary materials for this article are available online.

Hierarchical Back-off Modeling of Hiero Grammar based on Non-parametric Bayesian Model

In hierarchical phrase-based machine translation, a rule table is automatically learned by heuristically extracting synchronous rules from a parallel corpus. As a result, spuriously many rules are extracted which may be composed of various incorrect rules. The larger rule table incurs more run time for decoding and may result in lower translation quality. To resolve the problems, we propose a hierarchical back-off model for Hiero grammar, an instance of a synchronous context free grammar (SCFG), on the basis of the hierarchical Pitman-Yor process. The model can extract a compact rule and phrase table without resorting to any heuristics by hierarchically backing off to smaller phrases under SCFG. Inference is efficiently carried out using two-step synchronous parsing of Xiao et al., (2012) combined with slice sampling. In our experiments, the proposed model achieved higher or at least comparable translation quality against a previous Bayesian model on various language pairs; German/French/Spanish/JapaneseEnglish. When compared against heuristic models, our model achieved comparable translation quality on a full size GermanEnglish language pair in Europarl v7 corpus with significantly smaller grammar size; less than 10% of that for heuristic model.

Online Learning of Hierarchical Pitman-Yor Process Mixture of Generalized Dirichlet Distributions With Feature Selection.

In this paper, a novel statistical generative model based on hierarchical Pitman-Yor process and generalized Dirichlet distributions (GDs) is presented. The proposed model allows us to perform joint clustering and feature selection thanks to the interesting properties of the GD distribution. We develop an online variational inference algorithm, formulated in terms of the minimization of a Kullback-Leibler divergence, of our resulting model that tackles the problem of learning from high-dimensional examples. This variational Bayes formulation allows simultaneously estimating the parameters, determining the model's complexity, and selecting the appropriate relevant features for the clustering structure. Moreover, the proposed online learning algorithm allows data instances to be processed in a sequential manner, which is critical for large-scale and real-time applications. Experiments conducted using challenging applications, namely, scene recognition and video segmentation, where our approach is viewed as an unsupervised technique for visual learning in high-dimensional spaces, showed that the proposed approach is suitable and promising.

Latent IBP Compound Dirichlet Allocation.

We introduce the four-parameter IBP compound Dirichlet process (ICDP), a stochastic process that generates sparse non-negative vectors with potentially an unbounded number of entries. If we repeatedly sample from the ICDP we can generate sparse matrices with an infinite number of columns and power-law characteristics. We apply the four-parameter ICDP to sparse nonparametric topic modelling to account for the very large number of topics present in large text corpora and the power-law distribution of the vocabulary of natural languages. The model, which we call latent IBP compound Dirichlet allocation (LIDA), allows for power-law distributions, both, in the number of topics summarising the documents and in the number of words defining each topic. It can be interpreted as a sparse variant of the hierarchical Pitman-Yor process when applied to topic modelling. We derive an efficient and simple collapsed Gibbs sampler closely related to the collapsed Gibbs sampler of latent Dirichlet allocation (LDA), making the model applicable in a wide range of domains. Our nonparametric Bayesian topic model compares favourably to the widely used hierarchical Dirichlet process and its heavy tailed version, the hierarchical Pitman-Yor process, on benchmark corpora. Experiments demonstrate that accounting for the power-distribution of real data is beneficial and that sparsity provides more interpretable results.

Genre-Based Music Language Modeling with Latent Hierarchical Pitman-Yor Process Allocation

In this work we present a new Bayesian topic model: latent hierarchical Pitman-Yor process allocation (LHPYA), which uses hierarchical Pitman-Yor process priors for both word and topic distributions, and generalizes a few of the existing topic models, including the latent Dirichlet allocation (LDA), the bigram topic model and the hierarchical Pitman-Yor topic model. Using such priors allows for integration of n-grams with a topic model, while smoothing them with the state-of-the-art method. Our model is evaluated by measuring its perplexity on a dataset of musical genre and harmony annotations 3 Genre Database (3GDB) and by measuring its ability to predict musical genre from chord sequences. In terms of perplexity, for a 262-chord dictionary we achieve a value of 2.74, compared to 18.05 for trigrams and 7.73 for a unigram topic model. In terms of genre prediction accuracy with 9 genres, the proposed approach performs about 33% better in relative terms than genre-dependent n-grams, achieving 60.4% of accuracy.

Proteome Coverage Prediction for Integrated Proteomics Datasets

Comprehensive characterization of a proteome defines a fundamental goal in proteomics. In order to maximize proteome coverage for a complex protein mixture, i.e., to identify as many proteins as possible, various different fractionation experiments are typically performed and the individual fractions are subjected to mass spectrometric analysis. The resulting data are integrated into large and heterogeneous datasets. Proteome coverage prediction refers to the task of extrapolating the number of protein discoveries by future measurements conditioned on a sequence of already performed measurements. Proteome coverage prediction at an early stage enables experimentalists to design and plan efficient proteomics studies. To date, there does not exist any method that reliably predicts proteome coverage from integrated datasets. We present a generalized hierarchical Pitman-Yor process model that explicitly captures the redundancy within integrated datasets. The accuracy of our approach for proteome coverage prediction is assessed by applying it to an integrated proteomics dataset for the bacterium L. interrogans. The proposed procedure outperforms ad hoc extrapolation methods and prediction methods designed for non-integrated datasets. Furthermore, the maximally achievable proteome coverage is estimated for the experimental setup underlying the L. interrogans dataset. We discuss the implications of our results for determining rational stop criteria and their influence on the design of efficient and reliable proteomics studies.