Bayesian Nonparametric Model Research Articles

A generalization of hierarchical exchangeability on trees to directed acyclic graphs

Motivated by the problem of designing inference-friendly Bayesian nonparametric models in probabilistic programming languages, we introduce a general class of partially exchangeable random arrays which generalizes the notion of hierarchical exchangeability introduced in Austin and Panchenko (2014). We say that our partially exchangeable arrays are DAG-exchangeable since their partially exchangeable structure is governed by a collection of Directed Acyclic Graphs. More specifically, such a random array is indexed by ℕ |V| for some DAG G=(V,E), and its exchangeability structure is governed by the edge set E. We prove a representation theorem for such arrays which generalizes the Aldous-Hoover and Austin–Panchenko representation theorems.

Bayesian Non-Parametric Factor Analysis for Longitudinal Spatial Surfaces

We introduce a Bayesian non-parametric spatial factor analysis model with spatial dependency induced through a prior on factor loadings. For each column of the loadings matrix, spatial dependency is encoded using a probit stick-breaking process (PSBP) and a multiplicative gamma process shrinkage prior is used across columns to adaptively determine the number of latent factors. By encoding spatial information into the loadings matrix, meaningful factors are learned that respect the observed neighborhood dependencies, making them useful for assessing rates over space. Furthermore, the spatial PSBP prior can be used for clustering temporal trends, allowing users to identify regions within the spatial domain with similar temporal trajectories, an important task in many applied settings. In the manuscript, we illustrate the model’s performance in simulated data, but also in two real-world examples: longitudinal monitoring of glaucoma and malaria surveillance across the Peruvian Amazon. The R package spBFA, available on CRAN, implements the method.

A Gaussian Process Model of Cross-Category Dynamics in Brand Choice

Understanding individual customers’ sensitivities to prices, promotions, brand, and other aspects of the marketing mix is fundamental to a wide swath of marketing problems, including targeting and pricing. Companies that operate across many product categories have a unique opportunity, insofar as they can use purchasing data from one category to augment their insights in another. Such cross-category insights are especially crucial in situations where purchasing data may be rich in one category, and scarce in another. An important aspect of how consumers behave across categories is dynamics: preferences are not stable over time, and changes in individual-level preference parameters in one category may be indicative of changes in other categories, especially if those changes are driven by external factors. Yet, despite the rich history of modeling cross-category preferences, the marketing literature lacks a framework that flexibly accounts for correlated dynamics, or the cross-category interlinkages of individual-level sensitivity dynamics. In this work, we propose such a framework, leveraging individual-level, latent, multi-output Gaussian processes to build a non-parametric Bayesian choice model that allows information sharing of preference parameters across customers, time, and categories. We apply our model to grocery purchase data, and show that our model detects interesting dynamics of customers’ price sensitivities across multiple categories. Managerially, we show that capturing correlated dynamics yields substantial predictive gains, relative to benchmarks. Moreover, we find that capturing correlated dynamics can have implications for understanding changes in consumers preferences over time, and developing targeted marketing strategies based on those dynamics.

Methodology for using a Bayesian nonparametric model to uncover universal patterns in color naming

Language is an integral part of society which enables communication among its members. To shed light on how words gain their meaning and how their meaning evolves over time, color naming is often used as a case study. The color domain can be defined by a physical space, making it a useful concept for studying denotation of meaning. Though humans can distinguish millions of colors, language provides us with a small, manageable set of terms for categorizing the space. Partitions of the color space vary across different language groups and evolve over time (e.g. new color terms may enter a language). Investigating universal patterns in color naming provides insight into the mechanisms that give rise to the observed data. Recently, computational techniques have been utilized to study this phenomenon. Here, we develop a methodology for transforming a color naming data set—namely, the World Color Survey—which is based on constraints imposed by the stimulus space. This transformed data is used to initialize a nonparametric Bayesian machine learning model in order to implement a culture and theory-independent study of universal color naming patterns across different language groups. All of the methods described are executed by our Python software package called ColorBBDP.• Data from the World Color Survey is transformed from its original format into binary features vectors which can be given as input to the Beta-Bernoulli Dirichlet Process Mixture Model.• This paper provides a specific application of Variational Inference on the Beta-Bernoulli Dirichlet Process Mixture Model towards a color naming data set.• New mathematical measures for performing post-cluster analyses are also detailed in this paper.

Bayesian nonparametric disclosure risk assessment

Any decision about the release of microdata for public use is supported by the estimation of measures of disclosure risk, the most popular being the number τ1 of sample uniques that are also population uniques. In such a context, parametric and nonparametric partition-based models have been shown to have: i) the strength of leading to estimators of τ1 with desirable features, including ease of implementation, computational efficiency and scalability to massive data; ii) the weakness of producing underestimates of τ1 in realistic scenarios, with the underestimation getting worse as the tail behaviour of the empirical distribution of microdata gets heavier. To fix this underestimation phenomenon, we propose a Bayesian nonparametric partition-based model that can be tuned to the tail behaviour of the empirical distribution of microdata. Our model relies on the Pitman–Yor process prior, and it leads to a novel estimator of τ1 with all the desirable features of partition-based estimators and that, in addition, allows to reduce underestimation by tuning a “discount” parameter. We show the effectiveness of our estimator through its application to synthetic data and real data.

Probabilistic Assessment of Impact of Flexible Loads Under Network Tariffs in Low-voltage Distribution Networks

Given the historically static nature of low-voltage networks, distribution network companies do not possess the tools for dealing with an increasingly variable demand due to the high penetration of distributed energy resources (DERs). Within this context, this paper proposes a probabilistic framework for tariff design that minimises the impact of DER on network performance, stabilises the revenue of network company, and improves the equity of network cost allocation. To deal with the lack of customers' response, we also show how DER-specific tariffs can be complemented with an automated home energy management system (HEMS) that reduces peak demand while retaining the desired comfort level. The proposed framework comprises a nonparametric Bayesian model which statistically generates synthetic load and PV traces, a hot-water-use statistical model, a novel HEMS to schedule customers' controllable devices, and a probabilistic power flow model. Test cases using both energy- and demand-based network tariffs show that flat tariffs with a peak demand component reduce the customers' cost, and alleviate network constraints. This demonstrates, firstly, the efficacy of the proposed tool for the development of tariffs that are beneficial for the networks with a high penetration of DERs, and secondly, how customers' HEM systems can be part of the solution.

BNPmix: An R Package for Bayesian Nonparametric Modeling via Pitman-Yor Mixtures

BNPmix is an R package for Bayesian nonparametric multivariate density estimation, clustering, and regression, using Pitman-Yor mixture models, a flexible and robust generalization of the popular class of Dirichlet process mixture models. A variety of model specifications and state-of-the-art posterior samplers are implemented. In order to achieve computational efficiency, all sampling methods are written in C++ and seamless integrated into R by means of the Rcpp and RcppArmadillo packages. BNPmix exploits the ggplot2 capabilities and implements a series of generic functions to plot and print summaries of posterior densities and induced clustering of the data.

Truncated simulation and inference in edge-exchangeable networks

Edge-exchangeable probabilistic network models generate edges as an i.i.d.~sequence from a discrete measure, providing a simple means for statistical inference of latent network properties. The measure is often constructed using the self-product of a realization from a Bayesian nonparametric (BNP) discrete prior; but unlike in standard BNP models, the self-product measure prior is not conjugate the likelihood, hindering the development of exact simulation and inference algorithms. Approximation via finite truncation of the discrete measure is a straightforward alternative, but incurs an unknown approximation error. In this paper, we develop methods for forward simulation and posterior inference in random self-product-measure models based on truncation, and provide theoretical guarantees on the quality of the results as a function of the truncation level. The techniques we present are general and extend to the broader class of discrete Bayesian nonparametric models.

A Bayesian nonparametric mixture model for studying universal patterns in color naming

Variational Inference for the Beta-Bernoulli Dirichlet Process Mixture Model is employed to uncover universal patterns in color naming systems. The data used consist of 2552 participants from 106 World Color Survey languages. To study these languages collectively, the model is informed by universal biological, linguistic, and topological features of the task. We find that the majority of the naming systems are represented by eighteen clusters, each constituting a universal pattern. Novel mathematical techniques are developed to study the levels of similarity, underlying consensus, and diversity among these patterns. This implementation of nonparametric models demonstrates how machine learning methods can be tailored for behavioral science applications.

Health-Behaviors Associated With the Growing Risk of Adolescent Suicide Attempts: A Data-Driven Cross-Sectional Study.

Identify and examine the associations between health behaviors and increased risk of adolescent suicide attempts, while controlling for socio-economic and demographic differences. A data-driven analysis using cross-sectional data. Communities in the state of Montana from 1999 to 2017. Selected Montana as it persistently ranks among the top 3 vulnerable states in the U.S. over the past years. Selected 22,447 adolescents of whom 1,631 adolescents attempted suicide at least once. Overall 29 variables (predictors) accounting for psychological behaviors, illegal substances consumption, daily activities at schools and demographic backgrounds were considered. A library of machine learning algorithms along with the traditionally-used logistic regression were used to model and predict suicide attempt risk. Model performances-goodness-of-fit and predictive accuracy-were measured using accuracy, precision, recall and F-score metrics. Additionally, χ2 analysis was used to evaluate the statistical significance of each variable. The non-parametric Bayesian tree ensemble model outperformed all other models, with 80.0% accuracy in goodness-of-fit (F-score: 0.802) and 78.2% in predictive accuracy (F-score: 0.785). Key health-behaviors identified include: being sad/hopeless (p < 0.0001), followed by safety concerns at school (p < 0.0001), physical fighting (p < 0.0001), inhalant usage (p < 0.0001), illegal drugs consumption at school (p < 0.0001), current cigarette usage (p < 0.0001), and having first sex at an early age (below 15 years of age). Additionally, the minority groups (American Indian/Alaska Natives, Hispanics/Latinos) (p < 0.0001), and females (p < 0.0001) are also found to be highly vulnerable to attempting suicides. Significant contribution of this work is understanding the key health-behaviors and health disparities that lead to higher frequency of suicide attempts among adolescents, while accounting for the non-linearity and complex interactions among the outcome and the exposure variables. Findings provide insights on key health-behaviors that can be viewed as early warning signs/precursors of suicide attempts among adolescents.

Investigating alignment interpretability for low-resource NMT

The attention mechanism in Neural Machine Translation (NMT) models added flexibility to translation systems, and the possibility to visualize soft-alignments between source and target representations. While there is much debate about the relationship between attention and the yielded output for neural models (Jain and Wallace 2019; Serrano and Smith 2019; Wiegreffe and Pinter 2019; Vashishth et al. 2019), in this paper we propose a different assessment, investigating soft-alignment interpretability in low-resource scenarios. We experimented with different architectures (RNN (Bahdanau et al. 2015), 2D-CNN (Elbayad et al. 2018), and Transformer (Vaswani et al. 2017)), comparing them with regards to their ability to produce directly exploitable alignments. For evaluating exploitability, we replicated the Unsupervised Word Segmentation (UWS) task from Godard et al. (2018). There, source words are translated into unsegmented phone sequences. Posterior to training, the resulting soft-alignments are used for producing segmentation over the target side. Our results showed that a RNN-based NMT model produced the most exploitable alignments in this scenario. We then investigated methods for increasing its UWS scores by comparing the following methodologies: monolingual pre-training, input representation augmentation (hybrid model), and explicit word length optimization during training. We reached the best results by using the hybrid model, which uses an intermediate monolingual-rooted segmentation from a non-parametric Bayesian model (Goldwater 2007) to enrich the input representation before training.

A Bayesian non-parametric mixed-effects model of microbial growth curves.

Substantive changes in gene expression, metabolism, and the proteome are manifested in overall changes in microbial population growth. Quantifying how microbes grow is therefore fundamental to areas such as genetics, bioengineering, and food safety. Traditional parametric growth curve models capture the population growth behavior through a set of summarizing parameters. However, estimation of these parameters from data is confounded by random effects such as experimental variability, batch effects or differences in experimental material. A systematic statistical method to identify and correct for such confounding effects in population growth data is not currently available. Further, our previous work has demonstrated that parametric models are insufficient to explain and predict microbial response under non-standard growth conditions. Here we develop a hierarchical Bayesian non-parametric model of population growth that identifies the latent growth behavior and response to perturbation, while simultaneously correcting for random effects in the data. This model enables more accurate estimates of the biological effect of interest, while better accounting for the uncertainty due to technical variation. Additionally, modeling hierarchical variation provides estimates of the relative impact of various confounding effects on measured population growth.

A mixtures approach to assessing relationships between environmental and social determinants of health and neonatal size and body composition in the Healthy Start cohort

Background: Previous studies have identified neighborhood-level environmental and social exposures associated with birth weight, but no studies have quantified the effects of simultaneous exposures across these domains. Here we used Bayesian Kernel Machine Regression (BKMR) to estimate the independent and joint effects of environmental and social exposures on birth weight in the Healthy Start cohort in Denver, CO.Methods: The initial selection of exposures was guided by an environmental justice screening tool developed for California. Data on neighborhood-level environmental (e.g., fine particulate matter, traffic counts, impervious surfaces) and social (e.g., hospitalization rates, educational attainment, and crime rates) exposures were assigned to pregnant mothers based on their census tract at enrollment. Birth weight was abstracted from medical records. We used a two-step approach to fit our model. First, we used a Bayesian nonparametric variable selection model to reduce the number of candidate exposures. Second, we fit a BKMR model with the exposures, interaction terms for all exposure pairs, and individual-level covariates.Results: We had complete data on birth weight, exposures, and covariates for 792 infants. Out of 19 exposures originally considered, the non-parametric Bayesian model selected 9 (3 environmental and 6 social). BKMR results showed a mixture with all exposures above the median was associated with decreased birth weight. The mixture components selected most frequently by the model were respiratory hospitalization rate and the percentage of households in poverty. Respiratory hospitalization rates had the greatest effect on birth weight (β = -0.03, SE = 0.06) when all other exposures were held at the 75th percentile. For some exposures (e.g., fine particulate matter), relationships with birth weight appeared non-linear.Conclusions: We identified joint effects of environmental and social exposures on birth weight. Results suggest that analyses within one domain may limit the identification of important interactions that are associated with neonatal outcomes.

A nonparametric Bayesian model for estimating spectral densities of resting-state EEG twin data.

Electroencephalography (EEG) is a noninvasive neuroimaging modality that captures electrical brain activity many times per second. We seek to estimate power spectra from EEG data that ware gathered for 557 adolescent twin pairs through the Minnesota Twin Family Study (MTFS). Typically, spectral analysis methods treat time series from each subject separately, and independent spectral densities are fit to each time series. Since the EEG data were collected on twins, it is reasonable to assume that the time series have similar underlying characteristics, so borrowing information across subjects can significantly improve estimation. We propose a Nested Bernstein Dirichlet prior model to estimate the power spectrum of the EEG signal for each subject by smoothing periodograms within and across subjects while requiring minimal user input to tuning parameters. Furthermore, we leverage the MTFS twin study design to estimate the heritability of EEG power spectra with the hopes of establishing new endophenotypes. Through simulation studies designed to mimic the MTFS, we show our method out-performs a set of other popularmethods.

Bayesian mixture models for cytometry data analysis

AbstractBayesian mixture models are increasingly used for model‐based clustering and the follow‐up analysis on the clusters identified. As such, they are of particular interest for analyzing cytometry data where unsupervised clustering and association studies are often part of the scientific questions. Cytometry data are large quantitative data measured in a multidimensional space that typically ranges from a few dimensions to several dozens, and which keeps increasing due to innovative high‐throughput biotechonologies. We present several recent parametric and nonparametric Bayesian mixture modeling approaches, and describe advantages and limitations of these models under different research context for cytometry data analysis. We also acknowledge current computational challenges associated with the use of Bayesian mixture models for analyzing cytometry data, and we draw attention to recent developments in advanced numerical algorithms for estimating large Bayesian mixture models, which we believe have the potential to make Bayesian mixture model more applicable to new types of single‐cell data with higher dimensions.This article is categorized under: Statistical Learning and Exploratory Methods of the Data Sciences &gt; Knowledge Discovery Statistical Learning and Exploratory Methods of the Data Sciences &gt; Modeling Methods Statistical and Graphical Methods of Data Analysis &gt; Bayesian Methods and Theory

A dependent Dirichlet process model for survival data with competing risks

In this paper, we first propose a dependent Dirichlet process (DDP) model using a mixture of Weibull models with each mixture component resembling a Cox model for survival data. We then build a Dirichlet process mixture model for competing risks data without regression covariates. Next we extend this model to a DDP model for competing risks regression data by using a multiplicative covariate effect on subdistribution hazards in the mixture components. Though built on proportional hazards (or subdistribution hazards) models, the proposed nonparametric Bayesian regression models do not require the assumption of constant hazard (or subdistribution hazard) ratio. An external time-dependent covariate is also considered in the survival model. After describing the model, we discuss how both cause-specific and subdistribution hazard ratios can be estimated from the same nonparametric Bayesian model for competing risks regression. For use with the regression models proposed, we introduce an omnibus prior that is suitable when little external information is available about covariate effects. Finally we compare the models' performance with existing methods through simulations. We also illustrate the proposed competing risks regression model with data from a breast cancer study. An R package "DPWeibull" implementing all of the proposed methods is available at CRAN.

Single Satellite Imagery Superresolution Based on Hybrid Nonlocal Similarity Constrained Convolution Sparse Coding

The traditional superresolution methods based on image patches often ignore the consistency between the overlapped patches, causing block effects in produced images. The convolutional sparse coding based superresolution method uses the translation invariance of the convolution filter to directly encode the entire image, maintaining consistency and good performance. In this article, we propose a novel approach to single-image superresolution reconstruction based on hybrid nonlocal similarity constrained convolution sparse coding. We first decompose the input image into a smooth part and a texture part. The Bayesian nonparametric model can use more prior information of the original image, so we replace the previous bicubic interpolation with this method to better reconstruct the residual high-frequency information in the smooth part. When reconstructing the texture part, this article proposes a nonlocal similarity constrained convolutional sparse coding method, which transforms the reconstruction of the texture part to minimize the convolution sparse coding noise of the feature maps and classifies the image patches in the search space by using the correlation coefficients as the structural information, avoiding unnecessary weight calculation. Several methods were tested on satellite images extensively. Both visual inspection and quantitative analysis results demonstrate that our method outperforms other state-of-the-art methods and increases noise immunity effectively.

Fast Search and Estimation of Bayesian Nonparametric Mixture Models Using a Classification Annealing EM Algorithm

Bayesian nonparametric (BNP) infinite-mixture models provide flexible and accurate density estimation, cluster analysis, and regression. However, for the posterior inference of such a model, MCMC algorithms are complex, often need to be tailor-made for different BNP priors, and are intractable for large datasets. We introduce a BNP classification annealing EM algorithm which employs importance sampling estimation. This new fast-search algorithm, for virtually any given BNP mixture model, can quickly and accurately calculate the posterior predictive density estimate (by posterior averaging) and the maximum a-posteriori clustering estimate (by simulated annealing), even for datasets containing millions of observations. The algorithm can handle a wide range of BNP priors because it primarily relies on the ability to generate prior samples. The algorithm can be fast because in each iteration, it performs a sampling step for the (missing) clustering of the data points, instead of a costly E-step; and then performs direct posterior calculations in the M-step, given the sampled (imputed) clustering. The new algorithm is illustrated and evaluated through BNP Gaussian mixture model analyses of benchmark simulated data and real datasets. MATLAB code for the new algorithm is provided in the supplementary materials. Supplementary materials for this article are available online.

A Bayesian Small Area Model with Dirichlet Processes on the Responses

Abstract Typically survey data have responses with gaps, outliers and ties, and the distributions of the responses might be skewed. Usually, in small area estimation, predictive inference is done using a two-stage Bayesian model with normality at both levels (responses and area means). This is the Scott-Smith (S-S) model and it may not be robust against these features. Another model that can be used to provide a more robust structure is the two-stage Dirichlet process mixture (DPM) model, which has independent normal distributions on the responses and a single Dirichlet process on the area means. However, this model does not accommodate gaps, outliers and ties in the survey data directly. Because this DPM model has a normal distribution on the responses, it is unlikely to be realized in practice, and this is the problem we tackle in this paper. Therefore, we propose a two-stage non-parametric Bayesian model with several independent Dirichlet processes at the first stage that represents the data, thereby accommodating some of the difficulties with survey data and permitting a more robust predictive inference. This model has a Gaussian (normal) distribution on the area means, and so we call it the DPG model. Therefore, the DPM model and the DPG model are essentially the opposite of each other and they are both different from the S-S model. Among the three models, the DPG model gives us the best head-start to accommodate the features of the survey data. For Bayesian predictive inference, we need to integrate two data sets, one with the responses and other with area sizes. An application on body mass index, which is integrated with census data, and a simulation study are used to compare the three models (S-S, DPM, DPG); we show that the DPG model might be preferred.

A Dirichlet Process Prior Approach for Covariate Selection.

The variable selection problem in general, and specifically for the ordinary linear regression model, is considered in the setup in which the number of covariates is large enough to prevent the exploration of all possible models. In this context, Gibbs-sampling is needed to perform stochastic model exploration to estimate, for instance, the model inclusion probability. We show that under a Bayesian non-parametric prior model for analyzing Gibbs-sampling output, the usual empirical estimator is just the asymptotic version of the expected posterior inclusion probability given the simulation output from Gibbs-sampling. Other posterior conditional estimators of inclusion probabilities can also be considered as related to the latent probabilities distributions on the model space which can be sampled given the observed Gibbs-sampling output. This paper will also compare, in this large model space setup the conventional prior approach against the non-local prior approach used to define the Bayes Factors for model selection. The approach is exposed along with simulation samples and also an application of modeling the Travel and Tourism factors all over the world.