Discrete Random Measures Research Articles

Normalised latent measure factor models

Abstract We propose a methodology for modelling and comparing probability distributions within a Bayesian nonparametric framework. Building on dependent normalised random measures, we consider a prior distribution for a collection of discrete random measures where each measure is a linear combination of a set of latent measures, interpretable as characteristic traits shared by different distributions, with positive random weights. The model is nonidentified and a method for postprocessing posterior samples to achieve identified inference is developed. This uses Riemannian optimisation to solve a nontrivial optimisation problem over a Lie group of matrices. The effectiveness of our approach is validated on simulated data and in two applications to two real-world data sets: school student test scores and personal incomes in California. Our approach leads to interesting insights for populations and easily interpretable posterior inference.

Finite-dimensional Discrete Random Structures and Bayesian Clustering

Discrete random probability measures stand out as effective tools for Bayesian clustering. The investigation in the area has been very lively, with a strong emphasis on nonparametric procedures based on either the Dirichlet process or on more flexible generalizations, such as the normalized random measures with independent increments (NRMI). The literature on finite-dimensional discrete priors is much more limited and mostly confined to the standard Dirichlet-multinomial model. While such a specification may be attractive due to conjugacy, it suffers from considerable limitations when it comes to addressing clustering problems. In order to overcome these, we introduce a novel class of priors that arise as the hierarchical compositions of finite-dimensional random discrete structures. Despite the analytical hurdles such a construction entails, we are able to characterize the induced random partition and determine explicit expressions of the associated urn scheme and of the posterior distribution. A detailed comparison with (infinite-dimensional) NRMIs is also provided: indeed, informative bounds for the discrepancy between the partition laws are obtained. Finally, the performance of our proposal over existing methods is assessed on a real application where we study a publicly available dataset from the Italian education system comprising the scores of a mandatory nationwide test.

Hierarchical Dirichlet process and relative entropy

The Hierarchical Dirichlet process is a discrete random measure serving as an important prior in Bayesian non-parametrics. It is motivated with the study of groups of clustered data. Each group is modelled through a level two Dirichlet process and all groups share the same base distribution which itself is a drawn from a level one Dirichlet process. It has two concentration parameters with one at each level. The main results of the paper are the law of large numbers and large deviations for the hierarchical Dirichlet process and its mass when both concentration parameters converge to infinity. The large deviation rate functions are identified explicitly. The rate function for the hierarchical Dirichlet process consists of two terms corresponding to the relative entropies at each level. It is less than the rate function for the Dirichlet process, which reflects the fact that the number of clusters under the hierarchical Dirichlet process has a slower growth rate than under the Dirichlet process.

Asymptotic behavior of the number of distinct values in a sample from the geometric stick-breaking process

Discrete random probability measures are a key ingredient of Bayesian nonparametric inference. A sample generates ties with positive probability and a fundamental object of both theoretical and applied interest is the corresponding number of distinct values. The growth rate can be determined from the rate of decay of the small frequencies implying that, when the decreasingly ordered frequencies admit a tractable form, the asymptotics of the number of distinct values can be conveniently assessed. We focus on the geometric stick-breaking process and we investigate the effect of the distribution for the success probability on the asymptotic behavior of the number of distinct values. A whole range of logarithmic behaviors are obtained by appropriately tuning the prior. A two-term expansion is also derived and illustrated in a comparison with a larger family of discrete random probability measures having an additional parameter given by the scale of the negative binomial distribution.

Sampling hierarchies of discrete random structures

Hierarchical normalized discrete random measures identify a general class of priors that is suited to flexibly learn how the distribution of a response variable changes across groups of observations. A special case widely used in practice is the hierarchical Dirichlet process. Although current theory on hierarchies of nonparametric priors yields all relevant tools for drawing posterior inference, their implementation comes at a high computational cost. We fill this gap by proposing an approximation for a general class of hierarchical processes, which leads to an efficient conditional Gibbs sampling algorithm. The key idea consists of a deterministic truncation of the underlying random probability measures leading to a finite dimensional approximation of the original prior law. We provide both empirical and theoretical support for such a procedure.

On the inferential implications of decreasing weight structures in mixture models

Bayesian estimation of nonparametric mixture models strongly relies on available representations of discrete random probability measures. In particular, the order of the mixing weights plays an important role for the identifiability of component-specific parameters which, in turn, affects the convergence properties of posterior samplers. The geometric process mixture model provides a simple alternative to models based on the Dirichlet process that effectively addresses these issues. However, the rate of decay of the mixing weights for this model may be too fast for modeling data with a large number of components. The need for different decay rates arises. Some variants of the geometric process featuring different decay behaviors, while preserving the decreasing structure, are presented and investigated. An asymptotic characterization of the number of distinct values in a sample from the corresponding mixing measure is also given, highlighting the inferential implications of different prior specifications. The analysis is completed by a simulation study in the context of density estimation. It shows that by controlling the decaying rate, the mixture model is able to capture data with a large number of components.

Beta-Binomial stick-breaking non-parametric prior

A new class of nonparametric prior distributions, termed Beta-Binomial stick-breaking process, is proposed. By allowing the underlying length random variables to be dependent through a Beta marginals Markov chain, an appealing discrete random probability measure arises. The chain's dependence parameter controls the ordering of the stick-breaking weights, and thus tunes the model's label-switching ability. Also, by tuning this parameter, the resulting class contains the Dirichlet process and the Geometric process priors as particular cases, which is of interest for fast convergence of MCMC implementations. Some properties of the model are discussed and a density estimation algorithm is proposed and tested with simulated datasets.

Latent Nested Nonparametric Priors (with Discussion).

Discrete random structures are important tools in Bayesian nonparametrics and the resulting models have proven effective in density estimation, clustering, topic modeling and prediction, among others. In this paper, we consider nested processes and study the dependence structures they induce. Dependence ranges between homogeneity, corresponding to full exchangeability, and maximum heterogeneity, corresponding to (unconditional) independence across samples. The popular nested Dirichlet process is shown to degenerate to the fully exchangeable case when there are ties across samples at the observed or latent level. To overcome this drawback, inherent to nesting general discrete random measures, we introduce a novel class of latent nested processes. These are obtained by adding common and group-specific completely random measures and, then, normalizing to yield dependent random probability measures. We provide results on the partition distributions induced by latent nested processes, and develop a Markov Chain Monte Carlo sampler for Bayesian inferences. A test for distributional homogeneity across groups is obtained as a by-product. The results and their inferential implications are showcased on synthetic and real data.

A simple proof of Pitman–Yor’s Chinese restaurant process from its stick-breaking representation

Abstract For a long time, the Dirichlet process has been the gold standard discrete random measure in Bayesian nonparametrics. The Pitman-Yor process provides a simple and mathematically tractable generalization, allowing for a very flexible control of the clustering behaviour. Two commonly used representations of the Pitman-Yor process are the stick-breaking process and the Chinese restaurant process. The former is a constructive representation of the process which turns out very handy for practical implementation, while the latter describes the partition distribution induced. Obtaining one from the other is usually done indirectly with use of measure theory. In contrast, we propose here an elementary proof of Pitman-Yor’s Chinese Restaurant process from its stick-breaking representation.

Density regression using repulsive distributions

ABSTRACTFlexible regression is a traditional motivation for the development of non-parametric Bayesian models. A popular approach for this involves a joint model for responses and covariates, from which the desired result arises by conditioning on the covariates. Many such models involve the convolution of a continuous kernel with some discrete random probability measure defined as an infinite mixture of i.i.d. atoms. Following this strategy, we propose a flexible model that involves the concept of repulsion between atoms. We show that this results in a more parsimonious representation of the regression than the i.i.d. counterpart. The key aspect is that repulsion discourages mixture components that are near each other, thus favouring parsimony. We show that the conditional model retains the repulsive features, thus facilitating interpretation of the resulting flexible regression, and with little or no sacrifice of model fit compared to the infinite mixture case. We show the utility of the methodology by way of a small simulation study and an application to a well-known data set.

Dependent Species Sampling Models for Spatial Density Estimation

We consider a novel Bayesian nonparametric model for density estimation with an underlying spatial structure. The model is built on a class of species sampling models, which are discrete random probability measures that can be represented as a mixture of random support points and random weights. Specifically, we construct a collection of spatially dependent species sampling models and propose a mixture model based on this collection. The key idea is the introduction of spatial dependence by modeling the weights through a conditional autoregressive model. We present an extensive simulation study to compare the performance of the proposed model with competitors. The proposed model compares favorably to these alternatives. We apply the method to the estimation of summer precipitation density functions using Climate Prediction Center Merged Analysis of Precipitation data over East Asia.

An invariance principle for the two-dimensional parabolic Anderson model with small potential

We prove an invariance principle for the two-dimensional lattice parabolic Anderson model with small potential. As applications we deduce a Donsker type convergence result for a discrete random polymer measure, as well as a universality result for the spectrum of discrete random Schrodinger operators on large boxes with small potentials. Our proof is based on paracontrolled distributions and some basic results for multiple stochastic integrals of discrete martingales.

Are Gibbs-Type Priors the Most Natural Generalization of the Dirichlet Process?

Discrete random probability measures and the exchangeable random partitions they induce are key tools for addressing a variety of estimation and prediction problems in Bayesian inference. Here we focus on the family of Gibbs-type priors, a recent elegant generalization of the Dirichlet and the Pitman-Yor process priors. These random probability measures share properties that are appealing both from a theoretical and an applied point of view: (i) they admit an intuitive predictive characterization justifying their use in terms of a precise assumption on the learning mechanism; (ii) they stand out in terms of mathematical tractability; (iii) they include several interesting special cases besides the Dirichlet and the Pitman-Yor processes. The goal of our paper is to provide a systematic and unified treatment of Gibbs-type priors and highlight their implications for Bayesian nonparametric inference. We deal with their distributional properties, the resulting estimators, frequentist asymptotic validation and the construction of time-dependent versions. Applications, mainly concerning mixture models and species sampling, serve to convey the main ideas. The intuition inherent to this class of priors and the neat results they lead to make one wonder whether it actually represents the most natural generalization of the Dirichlet process.

On species sampling sequences induced by residual allocation models

We discuss fully Bayesian inference in a class of species sampling models that are induced by residual allocation (sometimes called stick-breaking) priors on almost surely discrete random measures. This class provides a generalization of the well-known Ewens sampling formula that allows for additional flexibility while retaining computational tractability. In particular, the procedure is used to derive the exchangeable predictive probability functions associated with the generalized Dirichlet process of Hjort (2000) and the probit stick-breaking prior of Chung and Dunson (2009) and Rodriguez and Dunson (2011). The procedure is illustrated with applications to genetics and nonparametric mixture modeling.

On a class of $$\sigma $$ σ -stable Poisson–Kingman models and an effective marginalized sampler

We investigate the use of a large class of discrete random probability measures, which is referred to as the class $$\mathcal {Q}$$Q, in the context of Bayesian nonparametric mixture modeling. The class $$\mathcal {Q}$$Q encompasses both the the two-parameter Poisson---Dirichlet process and the normalized generalized Gamma process, thus allowing us to comparatively study the inferential advantages of these two well-known nonparametric priors. Apart from a highly flexible parameterization, the distinguishing feature of the class $$\mathcal {Q}$$Q is the availability of a tractable posterior distribution. This feature, in turn, leads to derive an efficient marginal MCMC algorithm for posterior sampling within the framework of mixture models. We demonstrate the efficacy of our modeling framework on both one-dimensional and multi-dimensional datasets.

Modeling with Normalized Random Measure Mixture Models

The Dirichlet process mixture model and more general mixtures based on discrete random probability measures have been shown to be flexible and accurate models for density estimation and clustering. The goal of this paper is to illustrate the use of normalized random measures as mixing measures in nonparametric hierarchical mixture models and point out how possible computational issues can be successfully addressed. To this end, we first provide a concise and accessible introduction to normalized random measures with independent increments. Then, we explain in detail a particular way of sampling from the posterior using the Ferguson-Klass representation. We develop a thorough comparative analysis for location-scale mixtures that considers a set of alternatives for the mixture kernel and for the nonparametric component. Simulation results indicate that normalized random measure mixtures potentially represent a valid default choice for density estimation problems. As a byproduct of this study an R package to fit these models was produced and is available in the Comprehensive R Archive Network (CRAN).

Species sampling models: consistency for the number of species

This paper considers species sampling models using constructions that arise from Bayesian nonparametric prior distributions. A discrete random measure, used to generate a species sampling model, can have either a countable infinite number of atoms, which has been the emphasis in the recent literature, or a finite number of atoms K, while allowing K to be assigned a prior probability distribution on the positive integers. It is the latter class of model we consider here, due to the interpretation of K as the number of species. We demonstrate the consistency of the posterior distribution of K as the sample size increases.

Particle Filter Improved by Genetic Algorithm and Particle Swarm Optimization Algorithm

Particle filter algorithm is a filtering method which uses Monte Carlo idea within the framework of Bayesian estimation theory. It approximates the probability distribution by using particles and discrete random measure which is consisted of their weights, it updates new discrete random measure recursively according to the algorithm. When the sample is large enough, the discrete random measure approximates the true posteriori probability density function of the state variable. The particle filter algorithm is applicable to any non-linear non-Gaussian system. But the standard particle filter does not consider the current measured value, which will lead to particles with non-zero weights become less after some iterations, this results in particle degradation; re-sampling technique was used to inhibit degradation, but this will reduce the particle diversity, and results in particle impoverishment. To overcome the problems, this paper proposed a new particle filter which introduced genetic algorithm and particle swarm optimization algorithm. The new algorithm is called intelligent particle filter (IPF). Driving particles move to the optimal position by using particle swarm optimization algorithm, thus the numbers of effective particles was increased, the particle diversity was improved, and the particle degradation was inhibited. Replace the re-sampling method in traditional particle filter by using the choice, crossover and mutation operation of the genetic algorithm, avoiding the phenomenon of impoverishment. Simulation results show that the new algorithm improved the estimation accuracy significantly compare with the standard particle filter.

Effect on Prediction When Modeling Covariates in Bayesian Nonparametric Models

In biomedical research, it is often of interest to characterize biologic processes giving rise to observations and to make predictions of future observations. Bayesian nonparametric methods provide a means for carrying out Bayesian inference making as few assumptions about restrictive parametric models as possible. There are several proposals in the literature for extending Bayesian nonparametric models to include dependence on covariates. Limited attention, however, has been directed to the following two aspects. In this article, we examine the effect on fitting and predictive performance of incorporating covariates in a class of Bayesian nonparametric models by one of two primary ways: either in the weights or in the locations of a discrete random probability measure. We show that different strategies for incorporating continuous covariates in Bayesian nonparametric models can result in big differences when used for prediction, even though they lead to otherwise similar posterior inferences. When one needs the predictive density, as in optimal design, and this density is a mixture, it is better to make the weights depend on the covariates. We demonstrate these points via a simulated data example and in an application in which one wants to determine the optimal dose of an anticancer drug used in pediatric oncology.

A deletion-invariance property for random measures satisfying the Ghirlanda–Guerra identities

We show that if a discrete random measure on the unit ball of a separable Hilbert space satisfies the Ghirlanda–Guerra identities then by randomly deleting half of the points and renormalizing the weights of the remaining points we obtain the same random measure in distribution up to rotations.