Pitman Yor Research Articles

Bayesian mixture models (in)consistency for the number of clusters

AbstractBayesian nonparametric mixture models are common for modeling complex data. While these models are well‐suited for density estimation, recent results proved posterior inconsistency of the number of clusters when the true number of components is finite, for the Dirichlet process and Pitman–Yor process mixture models. We extend these results to additional Bayesian nonparametric priors such as Gibbs‐type processes and finite‐dimensional representations thereof. The latter include the Dirichlet multinomial process, the recently proposed Pitman–Yor, and normalized generalized gamma multinomial processes. We show that mixture models based on these processes are also inconsistent in the number of clusters and discuss possible solutions. Notably, we show that a postprocessing algorithm introduced for the Dirichlet process can be extended to more general models and provides a consistent method to estimate the number of components.

Joint modeling of causal phrases-sentiments-aspects using Hierarchical Pitman Yor Process

Traditional sentiment-aware topic models assume that topic or sentiment transition occurs from either a sentence to the next sentence or from a word to the next word. Such models cannot capture a topic or sentiment transition at phrase boundaries. Further, most of the models adopt a sentiment lexicon to initialize sentiment priors and this approach induces coverage problems. To overcome the above-cited limitations, we have proposed a topic model that extracts aspects, sentiments, and causal phrases simultaneously by leveraging Hierarchical Pitman Yor Process (HPYP) that is modified using a sentiment component, a word-tagger to guide the causal phrase generation and a sentiment prior initialized through a sequential model to address coverage problems. We have evaluated our model on six datasets and found that the proposed model outperforms the baselines in terms of perplexity by 14%, topical coherence by 20%, topic diversity by 5%, sentiment classification task’s accuracy by 4% and, precision, recall and F1 score by 2%. Ablation studies assert that sequence model based sentiment prior initialization results in increasing the accuracy of sentiment classification by 2%.

Truncated two‐parameter Poisson–Dirichlet approximation for Pitman–Yor process hierarchical models

AbstractIn this paper, we construct an approximation to the Pitman–Yor process by truncating its two‐parameter Poisson–Dirichlet representation. The truncation is based on a decreasing sequence of random weights, thus having a lower approximation error compared to the popular truncated stick‐breaking process. We develop an exact simulation algorithm to sample from the approximation process and provide an alternative MCMC algorithm for the parameter regime where the exact simulation algorithm becomes slow. The effectiveness of the simulation algorithms is demonstrated by the estimation of the functionals of a Pitman–Yor process. Then we adapt the approximation process into a Pitman–Yor process mixture model and devise a blocked Gibbs sampler for posterior inference.

Bayesian modeling via discrete nonparametric priors

The availability of complex-structured data has sparked new research directions in statistics and machine learning. Bayesian nonparametrics is at the forefront of this trend thanks to two crucial features: its coherent probabilistic framework, which naturally leads to principled prediction and uncertainty quantification, and its infinite-dimensionality, which exempts from parametric restrictions and ensures full modeling flexibility. In this paper, we provide a concise overview of Bayesian nonparametrics starting from its foundations and the Dirichlet process, the most popular nonparametric prior. We describe the use of the Dirichlet process in species discovery, density estimation, and clustering problems. Among the many generalizations of the Dirichlet process proposed in the literature, we single out the Pitman–Yor process, and compare it to the Dirichlet process. Their different features are showcased with real-data illustrations. Finally, we consider more complex data structures, which require dependent versions of these models. One of the most effective strategies to achieve this goal is represented by hierarchical constructions. We highlight the role of the dependence structure in the borrowing of information and illustrate its effectiveness on unbalanced datasets.

A Bayesian nonparametric method for detecting rapid changes in disease transmission

Whether an outbreak of infectious disease is likely to grow or dissipate is determined through the time-varying reproduction number, Rt. Real-time or retrospective identification of changes in Rt following the imposition or relaxation of interventions can thus contribute important evidence about disease transmission dynamics which can inform policymaking. Here, we present a method for estimating shifts in Rt within a renewal model framework. Our method, which we call EpiCluster, is a Bayesian nonparametric model based on the Pitman–Yor process. We assume that Rt is piecewise-constant, and the incidence data and priors determine when or whether Rt should change and how many times it should do so throughout the series. We also introduce a prior which induces sparsity over the number of changepoints. Being Bayesian, our approach yields a measure of uncertainty in Rt and its changepoints. EpiCluster is fast, straightforward to use, and we demonstrate that it provides automated detection of rapid changes in transmission, either in real-time or retrospectively, for synthetic data series where the Rt profile is known. We illustrate the practical utility of our method by fitting it to case data of outbreaks of COVID-19 in Australia and Hong Kong, where it finds changepoints coinciding with the imposition of non-pharmaceutical interventions. Bayesian nonparametric methods, such as ours, allow the volume and complexity of the data to dictate the number of parameters required to approximate the process and should find wide application in epidemiology. This manuscript was submitted as part of a theme issue on “Modelling COVID-19 and Preparedness for Future Pandemics”.

Hierarchical Dirichlet and Pitman–Yor process mixtures of shifted‐scaled Dirichlet distributions for proportional data modeling

AbstractIn this article, first, we propose a novel unsupervised learning method based on a hierarchical Dirichlet process mixture of shifted‐scaled Dirichlet (SSD) distributions. Second, we extend it to a hierarchical Pitman–Yor process mixture of SSD distributions. The goal is to find a model that properly fits complex real‐world data. Our models are based on SSD distributions that are more flexible than Dirichlet distribution in fitting proportional data. Simultaneous data fitting (parameter estimate) and model selection (model complexity determination) are possible with the suggested methods. We applied batch and online variational inference for learning the models. The online setting allows us to feed our models with large‐scale streaming data. The effectiveness of our proposed models is evaluated by four realistic and challenging applications, namely, spam email detection, texture clustering, traffic sign detection, and vehicle detection. Experimental results demonstrate the potential of our models to fit proportional data.

Importance conditional sampling for Pitman–Yor mixtures

Nonparametric mixture models based on the Pitman–Yor process represent a flexible tool for density estimation and clustering. Natural generalization of the popular class of Dirichlet process mixture models, they allow for more robust inference on the number of components characterizing the distribution of the data. We propose a new sampling strategy for such models, named importance conditional sampling (ICS), which combines appealing properties of existing methods, including easy interpretability and a within-iteration parallelizable structure. An extensive simulation study highlights the efficiency of the proposed method which, unlike other conditional samplers, shows stable performances for different specifications of the parameters characterizing the Pitman–Yor process. We further show that the ICS approach can be naturally extended to other classes of computationally demanding models, such as nonparametric mixture models for partially exchangeable data.

Bayesian Nonparametric Modeling for Predicting Dynamic Dependencies in Multiple Object Tracking.

The paper considers the problem of tracking an unknown and time-varying number of unlabeled moving objects using multiple unordered measurements with unknown association to the objects. The proposed tracking approach integrates Bayesian nonparametric modeling with Markov chain Monte Carlo methods to estimate the parameters of each object when present in the tracking scene. In particular, we adopt the dependent Dirichlet process (DDP) to learn the multiple object state prior by exploiting inherent dynamic dependencies in the state transition using the dynamic clustering property of the DDP. Using the DDP to draw the mixing measures, Dirichlet process mixtures are used to learn and assign each measurement to its associated object identity. The Bayesian posterior to estimate the target trajectories is efficiently implemented using a Gibbs sampler inference scheme. A second tracking approach is proposed that replaces the DDP with the dependent Pitman–Yor process in order to allow for a higher flexibility in clustering. The improved tracking performance of the new approaches is demonstrated by comparison to the generalized labeled multi-Bernoulli filter.

Learning (k,l)-context-sensitive probabilistic grammars with nonparametric Bayesian approach

Inferring formal grammars with nonparametric Bayesian approach is one of the most powerful approach for achieving high accuracy from unsupervised data. In this paper, mildly-context-sensitive probabilities, called (k, l)-context-sensitive probabilities, are defined on context-free grammars (CFGs). Inferring CFGs where the probabilities of rules are identified from contexts can be seen as a kind of dual approaches for distributional learning, in which the contexts characterize the substrings. We can handle the data sparsity for the context-sensitive probabilities by the smoothing effect of the hierarchical nonparametric Bayesian models such as Pitman–Yor processes (PYPs). We define the hierarchy of PYPs naturally by augmenting the infinite PCFGs. The blocked Gibbs sampling is known to be effective for inferring PCFGs. We show that, by modifying the inside probabilities, the blocked Gibbs sampling is able to be applied to the (k, l)-context-sensitive probabilistic grammars. At the same time, we show that the time complexity for (k, l)-context-sensitive probabilities of a CFG is \(O(|V|^{l+3}|w|^3)\) for each sentence w, where V is a set of nonterminals. Since it is computationally too expensive to iterate sufficient times especially when |V| is not small, some alternative sampling algorithms are required. Therefore, we propose a new sampling method called composite sampling, with which the sampling procedure is separated into sub-procedures for nonterminals and for derivation trees. Finally, we demonstrate that the inferred (k, 0)-context-sensitive probabilistic grammars can achieve lower perplexities than other probabilistic language models such as PCFGs, n-grams, and HMMs.

Pitman–Yor process mixture model for community structure exploration considering latent interaction patterns* *Project supported by Beijing Natural Science Foundation, China (Grant Nos. L181010 and 4172054), the National Key R&D Program of China (GrantNo. 2016YFB0801100), and the National Basic Research Program of China (Grant No. 2013CB329605).

The statistical model for community detection is a promising research area in network analysis. Most existing statistical models of community detection are designed for networks with a known type of community structure, but in many practical situations, the types of community structures are unknown. To cope with unknown community structures, diverse types should be considered in one model. We propose a model that incorporates the latent interaction pattern, which is regarded as the basis of constructions of diverse community structures by us. The interaction pattern can parameterize various types of community structures in one model. A collapsed Gibbs sampling inference is proposed to estimate the community assignments and other hyper-parameters. With the Pitman–Yor process as a prior, our model can automatically detect the numbers and sizes of communities without a known type of community structure beforehand. Via Bayesian inference, our model can detect some hidden interaction patterns that offer extra information for network analysis. Experiments on networks with diverse community structures demonstrate that our model outperforms four state-of-the-art models.

Clustering Species With Residual Covariance Matrix in Joint Species Distribution Models

Modeling species distributions over space and time is one of the major research topics in both ecology and conservation biology. Joint Species Distribution models (JSDMs) have recently been introduced as a tool to better model community data, by inferring a residual covariance matrix between species, after accounting for species' response to the environment. However, these models are computationally demanding, even when latent factors, a common tool for dimension reduction, are used. To address this issue, Taylor-Rodriguez et al. (2017) proposed to use a Dirichlet process, a Bayesian nonparametric prior, to further reduce model dimension by clustering species in the residual covariance matrix. Here, we built on this approach to include a prior knowledge on the potential number of clusters, and instead used a Pitman–Yor process to address some critical limitations of the Dirichlet process. We therefore propose a framework that includes prior knowledge in the residual covariance matrix, providing a tool to analyze clusters of species that share the same residual associations with respect to other species. We applied our methodology to a case study of plant communities in a protected area of the French Alps (the Bauges Regional Park), and demonstrated that our extensions improve dimension reduction and reveal additional information from the residual covariance matrix, notably showing how the estimated clusters are compatible with plant traits, endorsing their importance in shaping communities.

Bayesian nonparametric disclosure risk assessment

Any decision about the release of microdata for public use is supported by the estimation of measures of disclosure risk, the most popular being the number τ1 of sample uniques that are also population uniques. In such a context, parametric and nonparametric partition-based models have been shown to have: i) the strength of leading to estimators of τ1 with desirable features, including ease of implementation, computational efficiency and scalability to massive data; ii) the weakness of producing underestimates of τ1 in realistic scenarios, with the underestimation getting worse as the tail behaviour of the empirical distribution of microdata gets heavier. To fix this underestimation phenomenon, we propose a Bayesian nonparametric partition-based model that can be tuned to the tail behaviour of the empirical distribution of microdata. Our model relies on the Pitman–Yor process prior, and it leads to a novel estimator of τ1 with all the desirable features of partition-based estimators and that, in addition, allows to reduce underestimation by tuning a “discount” parameter. We show the effectiveness of our estimator through its application to synthetic data and real data.

Nonparametric Bayesian multiarmed bandits for single-cell experiment design

The problem of maximizing cell type discovery under budget constraints is a fundamental challenge for the collection and analysis of single-cell RNA-sequencing (scRNA-seq) data. In this paper we introduce a simple, computationally efficient and scalable Bayesian nonparametric sequential approach to optimize the budget allocation when designing a large-scale experiment for the collection of scRNA-seq data for the purpose of, but not limited to, creating cell atlases. Our approach relies on the following tools: (i) a hierarchical Pitman–Yor prior that recapitulates biological assumptions regarding cellular differentiation, and (ii) a Thompson sampling multiarmed bandit strategy that balances exploitation and exploration to prioritize experiments across a sequence of trials. Posterior inference is performed by using a sequential Monte Carlo approach which allows us to fully exploit the sequential nature of our species sampling problem. We empirically show that our approach outperforms state-of-the-art methods and achieves near-Oracle performance on simulated and scRNA-seq data alike. HPY-TS code is available at https://github.com/fedfer/HPYsinglecell.

Preserving Unique Structural Blocks of Targets in ISAR Imaging by Pitman–Yor Process

This article focuses on capturing and preserving unique structures in inverse synthetic aperture radar (ISAR) imaging procedure, such as structures with little quantity around the edges of targets, to further improve the performance of compressive sensing based ISAR imaging. Specifically, the newly proposed method utilizes the framework of extended block sparse Bayesian learning, and divides the two-dimension ISAR image into overlapping patches for capturing structures. Moreover, the method exploits Pitman-Yor process for learning and clustering space-variant local structures of the overlapping patches adaptively for preserving unique structures, since the power-law property of Pitman-Yor process helps to capture more unique variables in clustering procedure. Meanwhile, this article utilizes variational Bayesian inference for approximating the posterior of the hidden variables and estimating the parameters of the proposed model. Experiment results based on synthetic data, Electromagnetic simulated data, and real measured data, demonstrate the performance of the newly proposed method in obtaining ISAR images with high resolution using fewer measurements. And two metrics, correlation value and structural similarity, illuminate the performance of the method in preserving unique structures and weak scatterers in inherent structures in ISAR imaging.

Spherical data clustering and feature selection through nonparametric Bayesian mixture models with von Mises distributions

In this work, we tackle the problem of clustering spherical (i.e. L2 normalized) data vectors using nonparametric Bayesian mixture models with von Mises distributions. Our model is formulated by employing a nonparametric Bayesian framework known as the Pitman–Yor process mixture model. Different from finite mixture models in which the determination of the number of clusters is a crucial problem and often requires extra effort (e.g. by inspecting information criteria), the proposed model is nonparametric such that the number of clusters in the model is assumed to be infinite at the initial stage and will be inferred automatically based on the data. Moreover, an unsupervised feature selection scheme is incorporated into the proposed model to remove features that do not contribute significantly to the clustering process. We develop a stochastic variational inference algorithm to estimate model parameters, model complexity and feature saliencies simultaneously and effectively through the method of stochastic gradient ascent. We demonstrate the merits of the proposed nonparametric Bayesian mixture model on clustering spherical data vectors by conducting experiments on both synthetic datasets and two real-world applications namely topic novelty detection and flower images categorization.

The Pitman–Yor multinomial process for mixture modelling

SummaryDiscrete nonparametric priors play a central role in a variety of Bayesian procedures, most notably when used to model latent features, such as in clustering, mixtures and curve fitting. They are effective and well-developed tools, though their infinite dimensionality is unsuited to some applications. If one restricts to a finite-dimensional simplex, very little is known beyond the traditional Dirichlet multinomial process, which is mainly motivated by conjugacy. This paper introduces an alternative based on the Pitman–Yor process, which provides greater flexibility while preserving analytical tractability. Urn schemes and posterior characterizations are obtained in closed form, leading to exact sampling methods. In addition, the proposed approach can be used to accurately approximate the infinite-dimensional Pitman–Yor process, yielding improvements over existing truncation-based approaches. An application to convex mixture regression for quantitative risk assessment illustrates the theoretical results and compares our approach with existing methods.

Prior Sensitivity Analysis in a Semi-Parametric Integer-Valued Time Series Model.

We examine issues of prior sensitivity in a semi-parametric hierarchical extension of the INAR(p) model with innovation rates clustered according to a Pitman–Yor process placed at the top of the model hierarchy. Our main finding is a graphical criterion that guides the specification of the hyperparameters of the Pitman–Yor process base measure. We show how the discount and concentration parameters interact with the chosen base measure to yield a gain in terms of the robustness of the inferential results. The forecasting performance of the model is exemplified in the analysis of a time series of worldwide earthquake events, for which the new model outperforms the original INAR(p) model.

A simple proof of Pitman–Yor’s Chinese restaurant process from its stick-breaking representation

Abstract For a long time, the Dirichlet process has been the gold standard discrete random measure in Bayesian nonparametrics. The Pitman-Yor process provides a simple and mathematically tractable generalization, allowing for a very flexible control of the clustering behaviour. Two commonly used representations of the Pitman-Yor process are the stick-breaking process and the Chinese restaurant process. The former is a constructive representation of the process which turns out very handy for practical implementation, while the latter describes the partition distribution induced. Obtaining one from the other is usually done indirectly with use of measure theory. In contrast, we propose here an elementary proof of Pitman-Yor’s Chinese Restaurant process from its stick-breaking representation.

Distribution theory for hierarchical processes

Hierarchies of discrete probability measures are remarkably popular as nonparametric priors in applications, arguably due to two key properties: (i) they naturally represent multiple heterogeneous populations; (ii) they produce ties across populations, resulting in a shrinkage property often described as “sharing of information.” In this paper, we establish a distribution theory for hierarchical random measures that are generated via normalization, thus encompassing both the hierarchical Dirichlet and hierarchical Pitman–Yor processes. These results provide a probabilistic characterization of the induced (partially exchangeable) partition structure, including the distribution and the asymptotics of the number of partition sets, and a complete posterior characterization. They are obtained by representing hierarchical processes in terms of completely random measures, and by applying a novel technique for deriving the associated distributions. Moreover, they also serve as building blocks for new simulation algorithms, and we derive marginal and conditional algorithms for Bayesian inference.

Simultaneous clustering and feature selection via nonparametric Pitman–Yor process mixture models

Mixture models constitute one of the most important machine learning approaches. Indeed, they can be considered as the workhorse of generative machine learning. The majority of existing works consider mixtures of Gaussians. Unlike these works, this paper concentrates on nonparametric Bayesian models with Dirichlet-based mixtures. In particular, we consider the case when a Pitman–Yor process prior is adopted. Two central problems when considering such mixtures can be regarded as selecting ‘meaningful’ (or relevant) features and estimating the model’s parameters. We develop an efficient algorithm for model inference, based on the collapsed variational Bayes framework with 0th-order Taylor approximation. The merits and efficacy of the proposed nonparametric Bayesian model are demonstrated via challenging applications that concern real-world data clustering and 3D objects recognition.