Bayesian Nonparametrics Research Articles

Constraint-based learning for non-parametric continuous bayesian networks

Modeling high-dimensional multivariate distributions is a computationally challenging task. In the discrete case, Bayesian networks have been successfully used to reduce the complexity and to simplify the problem. However, they lack of a general model for continuous variables. In order to overcome this problem, Elidan (2010) proposed the model of copula Bayesian networks that parametrizes Bayesian networks using copula functions. We propose a new learning algorithm for this model based on a PC algorithm and a conditional independence test proposed by Bouezmarni et al. (2009). This test being non-parametric, no model assumptions are made allowing it to be as general as possible. This algorithm is compared on generated data with the parametric method proposed by Elidan (2010) and proves to have better results.

Bayesian nonparametrics for directional statistics

We introduce a density basis of the trigonometric polynomials that is suitable to mixture modelling. Statistical and geometric properties are derived, suggesting it as a circular analogue to the Bernstein polynomial densities. Nonparametric priors are constructed using this basis and a simulation study shows that the use of the resulting Bayes estimator may provide gains over comparable circular density estimators previously suggested in the literature.From a theoretical point of view, we propose a general prior specification framework for density estimation on compact metric space using sieve priors. This is tailored to density bases such as the one considered herein and may also be used to exploit their particular shape-preserving properties. Furthermore, strong posterior consistency is shown to hold under notably weak regularity assumptions and adaptive convergence rates are obtained in terms of the approximation properties of positive linear operators generating our models.

Probabilistic morphisms and Bayesian nonparametrics

In this paper we develop a functorial language of probabilistic morphisms and apply it to some basic problems in Bayesian nonparametrics. First we extend and unify the Kleisli category of probabilistic morphisms proposed by Lawvere and Giry with the category of statistical models proposed by Chentsov and Morse-Sacksteder. Then we introduce the notion of a Bayesian statistical model that formalizes the notion of a parameter space with a given prior distribution in Bayesian statistics. {We revisit the existence of a posterior distribution, using probabilistic morphisms}. In particular, we give an explicit formula for posterior distributions of the Bayesian statistical model, assuming that the underlying parameter space is a Souslin space and the sample space is a subset in a complete connected finite dimensional Riemannian manifold. Then we give a new proof of the existence of Dirichlet measures over any measurable space using a functorial property of the Dirichlet map constructed by Sethuraman.

Assessing the Impact of Precision Parameter Prior in Bayesian Non-parametric Growth Curve Modeling.

Bayesian non-parametric (BNP) modeling has been developed and proven to be a powerful tool to analyze messy data with complex structures. Despite the increasing popularity of BNP modeling, it also faces challenges. One challenge is the estimation of the precision parameter in the Dirichlet process mixtures. In this study, we focus on a BNP growth curve model and investigate how non-informative prior, weakly informative prior, accurate informative prior, and inaccurate informative prior affect the model convergence, parameter estimation, and computation time. A simulation study has been conducted. We conclude that the non-informative prior for the precision parameter is less preferred because it yields a much lower convergence rate, and growth curve parameter estimates are not sensitive to informative priors.

Bayesian Analysis of Japanese Pressurized Water Reactor Surveillance Data for Irradiation Embrittlement Prediction

Abstract The goal of this study was to identify the chemical component variables that should be used in irradiation embrittlement prediction and to determine the uncertainty of prediction of irradiation embrittlement of reactor pressure vessel (RPV) steels. To this end, statistical analysis using a Bayesian nonparametric (BNP) method was performed for Japanese pressurized water reactor (PWR) surveillance test data whose neutron fluence ranged from 3 × 1018 to 1.2 × 1020 n/cm2 (E > 1 MeV). The BNP method is a machine learning statistical method that takes the complexity and uncertainty of input variables into account. Statistical analysis using an index to select the most suitable combination of input variables revealed that four variables, namely, neutron fluence and Cu, Ni, and Si contents, were the most effective combination for embrittlement prediction. Cu content had the largest effect on the degree of embrittlement, followed by Ni and Si, in that order. The shift in the reference nil-ductility temperature (ΔRTNDT) was also calculated using the probability distribution obtained by the BNP method. The overall standard deviation of the residuals between the calculated and measured values of ΔRTNDT was 8.4 °C, which was comparable to that of the current Japanese embrittlement correlation method (JEAC4201-2013). The 95% credible interval (CI) of the posterior distribution of ΔRTNDT (i.e., the range in which data can exist when the uncertainty of input data is taken into consideration) calculated by the BNP method was identical to or smaller than the margin in the current Japanese embrittlement correlation method described in JEAC4201-2013. This result indicates that an adequate margin is provided in JEAC4201-2013.

More for less: predicting and maximizing genomic variant discovery via Bayesian nonparametrics

SummaryWhile the cost of sequencing genomes has decreased dramatically in recent years, this expense often remains nontrivial. Under a fixed budget, scientists face a natural trade-off between quantity and quality: spending resources to sequence a greater number of genomes or spending resources to sequence genomes with increased accuracy. Our goal is to find the optimal allocation of resources between quantity and quality. Optimizing resource allocation promises to reveal as many new variations in the genome as possible. We introduce a Bayesian nonparametric methodology to predict the number of new variants in a follow-up study based on a pilot study. When experimental conditions are kept constant between the pilot and follow-up, we find that our prediction is competitive with the best existing methods. Unlike current methods, though, our new method allows practitioners to change experimental conditions between the pilot and the follow-up. We demonstrate how this distinction allows our method to be used for more realistic predictions and for optimal allocation of a fixed budget between quality and quantity. We validate our method on cancer and human genomics data.

Nonparametric Bayesian background estimation for hyperspectral anomaly detection

We propose an anomaly detection algorithm based on nonparametric Bayesian (NB) background estimation for hyperspectral images. The background is modeled as a Gaussian mixture model (GMM) and the model parameters and order are estimated from the data in an unsupervised manner using nonparametric Bayesian methods called Chinese restaurant process mixtures (CRPM) and truncated Dirichlet process mixtures (TDPM). Since, in anomaly detection, the model parameters are estimated only using the background data in an unsupervised manner, we also propose to use a superpixel-based background data preparation method to obtain a slightly purified background data. In the proposed mixture models, the covariance matrix estimation problem is regularized by defining some conjugate priors to avoid obtaining an ill-conditioned covariance matrix. Finally, the GMM model whose parameters and model order are estimated by the proposed methods is used for anomaly detection. We compare the performances of the proposed methods with those of variational Bayesian, conventional GMM, and collaborative representation-based detector (CRD). Experiments on real hyperspectral data sets show that the proposed methods outperform the detection performance of the other methods.

Counting Fluorophores and Finding Potentials from FRET, Bayesian Nonparametrics for Challenging Problems in Biophysics

Fluorescence spectroscopy is a great way to reveal complex interactions at the molecular level, but because fluorophores are much smaller than the diffraction limit of light, one bright spot may contain many fluorescently labeled biomolecules. One technique to quantify the number of molecules within a diffraction limited spot is to illuminate the spot, putting the fluorophores into their fluorescently active bright states, then to watch for steplike brightness drops as the fluorophores photobleach one-by-one. This analysis is impeded by detector noise, shot noise, and by the fact that fluorophores can have dark states and multiple bright states. Furthermore, because the number of fluorophores is unknown, we cannot utilize traditional Bayesian methods to count and we must invoke new methods to solve the problem. Here we use Bayesian nonparametric (BNP) tools, that are necessary as the number of fluorophores is unknown, to solve the counting problem. We conclude that our method allows us to count upwards of 100 fluorophores. Single molecule Forster resonance energy transfer (FRET) based fluorescence experiments are a useful way to probe configuration dynamics of molecules based on the distance between donor and acceptor labels. Here we show that we can use single photon information to extract potential profiles from molecules of interest. We do so using Gaussian processes, a powerful tool within BNPs

A Bayesian Nonparametric Approach to Learning Molecular Species and Diffusion Dynamics Based on Photon Arrival Data

Understanding how life works requires the study of biochemical reactions in vivo on a single-molecule level, which typically involves monitoring molecular species and how they interconvert to each other. These studies should also be conducted with freely diffusing molecules since diffusion is essential in important processes such as transcription factors finding sites on a DNA. Our goal is to learn about the biochemical reaction rates and some diffusion dynamics of biomolecules at a single-photon level. Previous lab efforts have been focused on probing the number of molecular species, simultaneously and self-consistently, via directly conducting a photon-by-photon analysis on the experimental data in a nonparametric fashion without presuming the total number of chemical species. Here, we would like to go beyond this by taking transformations into account as well, and we will do so by leveraging the novel tool of Bayesian nonparametrics (BNP) that are now computationally feasible based on the methods from computational statistics that we are exploiting in the lab, including the Hamiltonian Monte Carlo (HMC), sequential Bayesian filtering, etc. Our approach has been validated using some simulated data containing roughly several thousand of photon arrival times, and it will soon be tested against the experimental data we have obtained recently.

Unsupervised Bayesian Non Parametric approach for Non-Intrusive Load Monitoring based on time of usage

Infinite Factorial Hidden Markov Model (iFHMM) is an attractive extension of Factorial Hidden Markov Model for Non-Intrusive Load Monitoring (NILM) which infers automatically the number of appliances in households and adapts its effective model complexity to fit the data. However, due to the infinite dimension nature of the model, its inference is difficult and faces several issues in the context of NILM. First, the model is hindered by computational complexity because it cannot deal with a number of appliances greater than ten. Second, it still requires accuracy improvement. Third, the model convergence may take a too long time. Therefore, a new infinite Factorial Hidden Markov Model constrained on Contextual features (iFHMMCC) is developed to overcome these shortcomings. To this end, appliances’ time of usage is added to the model in order improve disaggregation accuracy. Besides, it is used to alleviate the inference’s computational complexity and makes the model more tractable than Bayesian Non-Parametric (BNP) state of the art algorithms. Evaluation is performed on REDD database and the proposed approach is compared to five different well-known BNP disaggregation algorithms. The obtained results demonstrate an encouraging improvement in disaggregation accuracy as well as the inference’s computational complexity.

Indoor Multi-Speaker Localization Based on Bayesian Nonparametrics in the Circular Harmonic Domain

Circular microphone arrays have been used for multi-speaker localization in computational auditory scene analysis, for their high flexibility in sound field analysis, including the generation of frequency-invariant eigenbeams for wideband acoustic sources. However, the localization performance of existing circular harmonic approaches, such as circular harmonics beamformer (CHB) depends strongly on the physical characteristics (such as shape) of sensor arrays, and the level of uncertainties presented in acoustic environments (such as background noise, room reverberation, and the number of sources). These uncertainties may limit the performance or practical application of the speaker localization algorithms. To address these issues, in this paper, we present a new indoor multi-speaker localization method in the circular harmonic domain based on the acoustic holography beamforming (AHB) technique and the Bayesian nonparametrics (BNP) method. More specifically, we use the AHB technique, which combines the delay-and-sum beamforming with acoustic-holography-based virtual sensing, to generate direction of arrival (DOA) measurements in the time-frequency (TF) domain, and then design a BNP algorithm based on the infinite Gaussian mixture model (IGMM) to estimate the DOAs of the individual sources without the prior knowledge about the number of sources. These estimates may degrade in the presence of room reverberation and background noise. To address this issue, we develop a robust TF bin selection and permutation method on the basis of mixture weights, using power, power ratio and local variance estimated at each TF bin. Experiments performed on both simulated and real-data show that our method gives significantly better performance, than four recent baseline methods, in a variety of noise and reverberation levels, in terms of the root-mean-square error (RMSE) of the DOA estimation and the source detecting success rate.

Tensor RNN With Bayesian Nonparametric Mixture for Radar HRRP Modeling and Target Recognition

To deal with the temporal dependence between range cells in high resolution range profile (HRRP), dynamic methods, especially recurrent neural network (RNN), have been employed to extract features for target recognition. However, RNN has difficulty in complex and diverse sequence modeling problems as it ignores non-stationary sequential relationship between time-steps by sharing same parameters among all time-steps. Given this issue, we propose tensor recurrent neural network with Gaussian mixture model (GmTRNN) for HRRP, not only making use of temporal characteristic but also modeling the variation among its patterns. Specifically, a novel tensor RNN is developed by extending all the parameters in the form of tensor to explore diverse temporal dependence between range cells within an HRRP sample, where a mixture model is introduced to determine the parameters of each time-step in tensor RNN. Moreover, to take advantage of Bayesian nonparametrics in handling the unknown number of mixture components, we further propose the tensor recurrent neural network with Dirichlet process mixture (DPmTRNN). For scalable and joint training of clustering and recognition, we present effective hybrid online variational inference and stochastic gradient descent method. Experiments on benchmark data, measured and simulated HRRP data demonstrate the the effectiveness and efficiency of our models and its robustness to HRRP shift.

Truncated simulation and inference in edge-exchangeable networks

Edge-exchangeable probabilistic network models generate edges as an i.i.d.~sequence from a discrete measure, providing a simple means for statistical inference of latent network properties. The measure is often constructed using the self-product of a realization from a Bayesian nonparametric (BNP) discrete prior; but unlike in standard BNP models, the self-product measure prior is not conjugate the likelihood, hindering the development of exact simulation and inference algorithms. Approximation via finite truncation of the discrete measure is a straightforward alternative, but incurs an unknown approximation error. In this paper, we develop methods for forward simulation and posterior inference in random self-product-measure models based on truncation, and provide theoretical guarantees on the quality of the results as a function of the truncation level. The techniques we present are general and extend to the broader class of discrete Bayesian nonparametric models.

Variational posterior approximation using stochastic gradient ascent with adaptive stepsize

Scalable algorithms of variational posterior approximation allow Bayesian nonparametrics such as Dirichlet process mixture to scale up to larger dataset at fractional cost. Recent algorithms, notably the stochastic variational inference performs local learning from minibatch. The main problem with stochastic variational inference is that it relies on closed form solution. Stochastic gradient ascent is a modern approach to machine learning and is widely deployed in the training of deep neural networks. In this work, we explore using stochastic gradient ascent as a fast algorithm for the posterior approximation of Dirichlet process mixture. However, stochastic gradient ascent alone is not optimal for learning. In order to achieve both speed and performance, we turn our focus to stepsize optimization in stochastic gradient ascent. As as intermediate approach, we first optimize stepsize using the momentum method. Finally, we introduce Fisher information to allow adaptive stepsize in our posterior approximation. In the experiments, we justify that our approach using stochastic gradient ascent do not sacrifice performance for speed when compared to closed form coordinate ascent learning on these datasets. Lastly, our approach is also compatible with deep ConvNet features as well as scalable to large class datasets such as Caltech256 and SUN397.

Direct Photon-by-Photon Analysis of Time-Resolved Pulsed Excitation Data using Bayesian Nonparametrics

SUMMARYLifetimes of chemical species are typically estimated by either fitting time-correlated single-photon counting (TCSPC) histograms or phasor analysis from time-resolved photon arrivals. While both methods yield lifetimes in a computationally efficient manner, their performance is limited by choices made on the number of distinct chemical species contributing photons. However, the number of species is encoded in the photon arrival times collected for each illuminated spot and need not be set by hand a priori. Here, we propose a direct photon-by-photon analysis of data drawn from pulsed excitation experiments to infer, simultaneously and self-consistently, the number of species and their associated lifetimes from a few thousand photons. We do so by leveraging new mathematical tools within the Bayesian nonparametric. We benchmark our method for both simulated and experimental data for 1–4 species.

Parameter clustering in Bayesian functional principal component analysis of neuroscientific data.

The extraordinary advancements in neuroscientific technology for brain recordings over the last decades have led to increasingly complex spatiotemporal data sets. To reduce oversimplifications, new models have been developed to be able to identify meaningful patterns and new insights within a highly demanding data environment. To this extent, we propose a new model called parameter clustering functional principal component analysis (PCl-fPCA) that merges ideas from functional data analysis and Bayesian nonparametrics to obtain a flexible and computationally feasible signal reconstruction and exploration of spatiotemporal neuroscientific data. In particular, we use a Dirichlet process Gaussian mixture model to cluster functional principal component scores within the standard Bayesian functional PCA framework. This approach captures the spatial dependence structure among smoothed time series (curves) and its interaction with the time domain without imposing a prior spatial structure on the data. Moreover, by moving the mixture from data to functional principal component scores, we obtain a more general clustering procedure, thus allowing a higher level of intricate insight and understanding of the data. We present results from a simulation study showing improvements in curve and correlation reconstruction compared with different Bayesian and frequentist fPCA models and we apply our method to functional magnetic resonance imaging and electroencephalogram data analyses providing a rich exploration of the spatiotemporal dependence in brain time series.

Fast Search and Estimation of Bayesian Nonparametric Mixture Models Using a Classification Annealing EM Algorithm

Bayesian nonparametric (BNP) infinite-mixture models provide flexible and accurate density estimation, cluster analysis, and regression. However, for the posterior inference of such a model, MCMC algorithms are complex, often need to be tailor-made for different BNP priors, and are intractable for large datasets. We introduce a BNP classification annealing EM algorithm which employs importance sampling estimation. This new fast-search algorithm, for virtually any given BNP mixture model, can quickly and accurately calculate the posterior predictive density estimate (by posterior averaging) and the maximum a-posteriori clustering estimate (by simulated annealing), even for datasets containing millions of observations. The algorithm can handle a wide range of BNP priors because it primarily relies on the ability to generate prior samples. The algorithm can be fast because in each iteration, it performs a sampling step for the (missing) clustering of the data points, instead of a costly E-step; and then performs direct posterior calculations in the M-step, given the sampled (imputed) clustering. The new algorithm is illustrated and evaluated through BNP Gaussian mixture model analyses of benchmark simulated data and real datasets. MATLAB code for the new algorithm is provided in the supplementary materials. Supplementary materials for this article are available online.

Traffic-Aware Backscatter Communications in Wireless-Powered Heterogeneous Networks

With the emerging Internet-of-Things services, massive machine-to-machine (M2M) communication will be deployed on top of human-to-human (H2H) communication in the near future. Due to the coexistence of M2M and H2H communications, the performance of M2M (i.e., secondary) networks depends largely on the H2H (i.e., primary) network. In this paper, we propose ambient backscatter communication for the M2M networks which exploits the energy (signal) sources of the H2H network, referring to its traffic applications and popularity. In order to maximize the harvesting and transmission opportunities offered by varying traffic sources of the H2H network, we adopt a Bayesian nonparametric (BNP) learning algorithm to classify traffic applications (patterns) for secondary transmitters (STs). We then analyze the performance of STs using the stochastic geometric approach, based on a criterion for optimal traffic selection. Because of the mathematical intractability of the optimal criterion, we have attained a suboptimal traffic selection criterion which provides more tractable analysis. Results are presented to validate the performance of the proposed BNP classification algorithm and the criterion, as well as the impact of traffic sources and popularity.

Gaussbock: Fast Parallel-iterative Cosmological Parameter Estimation with Bayesian Nonparametrics

Abstract We present and apply Gaussbock, a new embarrassingly parallel iterative algorithm for cosmological parameter estimation designed for an era of cheap parallel-computing resources. Gaussbock uses Bayesian nonparametrics and truncated importance sampling to accurately draw samples from posterior distributions with an orders-of-magnitude speed-up in wall time over alternative methods. Contemporary problems in this area often suffer from both increased computational costs due to high-dimensional parameter spaces and consequent excessive time requirements, as well as the need to fine-tune proposal distributions or sampling parameters. Gaussbock is designed specifically with these issues in mind. We explore and validate the performance and convergence of the algorithm on a fast approximation to the Dark Energy Survey Year 1 (DES Y1) posterior, finding reasonable scaling behavior with the number of parameters. We then test on the full DES Y1 posterior using large-scale supercomputing facilities and recover reasonable agreement with previous chains, although the algorithm can underestimate the tails of poorly constrained parameters. Additionally, we discuss and demonstrate how Gaussbock recovers complex posterior shapes very well at lower dimensions, but faces challenges to perform well on such distributions in higher dimensions. In addition, we provide the community with a user-friendly software tool for accelerated cosmological parameter estimation based on the methodology described in this paper.

Modeling neuroaffective biomarkers of drug addiction: A Bayesian nonparametric approach using dirichlet process mixtures

BackgroundThe properties of neurophysiological processes related to addiction have received much attention in the literature. However, empirical evidence of meaningful and useful characterization of these processes is limited. Recent studies have found that electrophysiological responses to emotional and drug-related cues can be used to create profiles that reliably predict smoking relapse. New methodThis paper evaluates the validity of classifying electrophysiological responses into distinct profiles using a Bayesian dirichlet process mixture (DPM) model. The DPM is a Bayesian nonparametric (BNP) method to modeling unknown number of profiles characterized by uncertainty in cluster membership and in cluster number. ResultsThe DPM model confirmed previously identified neuroaffective reactivity profiles, but also revealed a finer level of granularity in the clustering. Specifically, in addition to the two clusters previously identified in the literature, the BNP methods identified a cluster of individuals showing similar responses to smoking, pleasant, neutral and unpleasant cues. Comparison with existing methodsBNP models provide an alternative to the k-mean clustering approach to modeling EEG-based neuroaffective profiles. Unlike k-means clustering, BNP models compute the probability that a subject belongs to a cluster while taking into consideration uncertainty in the number of clusters. ConclusionsOur results confirm the reliability of the two clusters previously identified in these data, but also provide new insights by revealing a cluster that presented similar responses to stimuli with different contents. This finding may be related to the uncertainty in classification or overlapping brain-reactivity profiles.