Blocked Gibbs Sampler Research Articles

A Bayesian approach for envelope models

The envelope model is a new paradigm to address estimation and prediction in multivariate analysis. Using sufficient dimension reduction techniques, it has the potential to achieve substantial efficiency gains compared to standard models. This model was first introduced by [Statist. Sinica 20 (2010) 927–960] for multivariate linear regression, and has since been adapted to many other contexts. However, a Bayesian approach for analyzing envelope models has not yet been investigated in the literature. In this paper, we develop a comprehensive Bayesian framework for estimation and model selection in envelope models in the context of multivariate linear regression. Our framework has the following attractive features. First, we use the matrix Bingham distribution to construct a prior on the orthogonal basis matrix of the envelope subspace. This prior respects the manifold structure of the envelope model, and can directly incorporate prior information about the envelope subspace through the specification of hyperparamaters. This feature has potential applications in the broader Bayesian sufficient dimension reduction area. Second, sampling from the resulting posterior distribution can be achieved by using a block Gibbs sampler with standard associated conditionals. This in turn facilitates computationally efficient estimation and model selection. Third, unlike the current frequentist approach, our approach can accommodate situations where the sample size is smaller than the number of responses. Lastly, the Bayesian approach inherently offers comprehensive uncertainty characterization through the posterior distribution. We illustrate the utility of our approach on simulated and real datasets.

Convergence analysis of block Gibbs samplers for Bayesian linear mixed models with $p>N$

Exploration of the intractable posterior distributions associated with Bayesian versions of the general linear mixed model is often performed using Markov chain Monte Carlo. In particular, if a conditionally conjugate prior is used, then there is a simple two-block Gibbs sampler available. Román and Hobert [Linear Algebra Appl. 473 (2015) 54–77] showed that, when the priors are proper and the $X$ matrix has full column rank, the Markov chains underlying these Gibbs samplers are nearly always geometrically ergodic. In this paper, Román and Hobert’s (2015) result is extended by allowing improper priors on the variance components, and, more importantly, by removing all assumptions on the $X$ matrix. So, not only is $X$ allowed to be (column) rank deficient, which provides additional flexibility in parameterizing the fixed effects, it is also allowed to have more columns than rows, which is necessary in the increasingly important situation where $p>N$. The full rank assumption on $X$ is at the heart of Román and Hobert’s (2015) proof. Consequently, the extension to unrestricted $X$ requires a substantially different analysis.

Nonparametric Bayesian Double Articulation Analyzer for Direct Language Acquisition From Continuous Speech Signals

Human infants can discover words directly from unsegmented speech signals without any explicitly labeled data. In this paper, we develop a novel machine learning method called nonparametric Bayesian double articulation analyzer (NPB-DAA) that can directly acquire language and acoustic models from observed continuous speech signals. For this purpose, we propose an integrative generative model that combines a language model and an acoustic model into a single generative model called the "hierarchical Dirichlet process hidden language model" (HDP-HLM). The HDP-HLM is obtained by extending the hierarchical Dirichlet process hidden semi-Markov model (HDP-HSMM) proposed by Johnson et al. An inference procedure for the HDP-HLM is derived using the blocked Gibbs sampler originally proposed for the HDP-HSMM. This procedure enables the simultaneous and direct inference of language and acoustic models from continuous speech signals. Based on the HDP-HLM and its inference procedure, we developed a novel double articulation analyzer. By assuming HDP-HLM as a generative model of observed time series data, and by inferring latent variables of the model, the method can analyze latent double articulation structure, i.e., hierarchically organized latent words and phonemes, of the data in an unsupervised manner. The novel unsupervised double articulation analyzer is called NPB-DAA. The NPB-DAA can automatically estimate double articulation structure embedded in speech signals. We also carried out two evaluation experiments using synthetic data and actual human continuous speech signals representing Japanese vowel sequences. In the word acquisition and phoneme categorization tasks, the NPB-DAA outperformed a conventional double articulation analyzer (DAA) and baseline automatic speech recognition system whose acoustic model was trained in a supervised manner.

Bayesian Hierarchical Linear Modeling of Profile Data With Applications to Quality Control of Nanomanufacturing

In this paper, we propose to use the multiple ultrasonic attenuation profiles measured at different locations of a specimen to infer the microstructure of metal-matrix nanocomposites. We present a general framework to connect the profile data with both explanatory variables and product quality parameters for quality inference. In particular, a hierarchical linear model with level-2 variance heterogeneity is proposed to model the relationship between ultrasonic attenuation profiles, ultrasonic frequency, and the microstructural parameters of the nanocomposites. An integrated Bayesian framework for model estimation, model selection, and inference of the microstructural parameters is proposed through blocked Gibbs sampling, intrinsic Bayes factor, and importance sampling. The effectiveness of the proposed approach is illustrated through case studies.

Bayesian analysis for confirmatory factor model with finite-dimensional Dirichlet prior mixing

ABSTRACTConfirmatory factor analysis (CFA) model is a useful multivariate statistical tool for interpreting relationships between latent variables and manifest variables. Often statistical results based on a single CFA are seriously distorted when data set takes on heterogeneity. To address the heterogeneity resulting from the multivariate responses, we propose a Bayesian semiparametric modeling for CFA. The approach relies on using a prior over the space of mixing distributions with finite components. Blocked Gibbs sampler is implemented to cope with the posterior analysis. Results obtained from a simulation study and a real data set are presented to illustrate the methodology.

Assessing Heterogeneity for Factor Analysis Model with Continuous and Ordinal Outcomes

Factor analysis models with continuous and ordinal responses are a useful tool for assessing relations between the latent variables and mixed observed responses. These models have been successfully applied to many different fields, including behavioral, educational, and social-psychological sciences. However, within the Bayesian analysis framework, most developments are constrained within parametric families, of which the particular distributions are specified for the parameters of interest. This leads to difficulty in dealing with outliers and/or distribution deviations. In this paper, we propose a Bayesian semiparametric modeling for factor analysis model with continuous and ordinal variables. A truncated stick-breaking prior is used to model the distributions of the intercept and/or covariance structural parameters. Bayesian posterior analysis is carried out through the simulation-based method. Blocked Gibbs sampler is implemented to draw observations from the complicated posterior. For model selection, the logarithm of pseudomarginal likelihood is developed to compare the competing models. Empirical results are presented to illustrate the application of the methodology.

Posterior sampling from $\varepsilon$-approximation of normalized completely random measure mixtures

This paper adopts a Bayesian nonparametric mixture model where the mixing distribution belongs to the wide class of normalized homogeneous completely random measures. We propose a truncation method for the mixing distribution by discarding the weights of the unnormalized measure smaller than a threshold. We prove convergence in law of our approximation, provide some theoretical properties, and characterize its posterior distribution so that a blocked Gibbs sampler is devised. The versatility of the approximation is illustrated by two different applications. In the first the normalized Bessel random measure, encompassing the Dirichlet process, is introduced; goodness of fit indexes show its good performances as mixing measure for density estimation. The second describes how to incorporate covariates in the support of the normalized measure, leading to a linear dependent model for regression and clustering.

Stick-Breaking Representation and Computation for Normalized Generalized Gamma Processes

Using fractions of gamma and exponentially titled stable random variables this article develops a stick-breaking representation of a truncated normalized generalized gamma process. Sampling from the posterior of this process requires sampling from gamma titled stable random variables and we develop an algorithm to do so that is readily implemented in the open source software R. A Blocked Gibbs sampling algorithm for a Bayesian kernel mixture model is then described and we compare the performance of our algorithm with an algorithm recently proposed in the literature.

Linear-Time Gibbs Sampling in Piecewise Graphical Models

Many real-world Bayesian inference problems such as preference learning or trader valuation modeling in financial markets naturally use piecewise likelihoods. Unfortunately, exact closed-form inference in the underlying Bayesian graphical models is intractable in the general case and existing approximation techniques provide few guarantees on both approximation quality and efficiency. While (Markov Chain) Monte Carlo methods provide an attractive asymptotically unbiased approximation approach, rejection sampling and Metropolis-Hastings both prove inefficient in practice, and analytical derivation of Gibbs samplers require exponential space and time in the amount of data. In this work, we show how to transform problematic piecewise likelihoods into equivalent mixture models and then provide a blocked Gibbs sampling approach for this transformed model that achieves an exponential-to-linear reduction in space and time compared to a conventional Gibbs sampler. This enables fast, asymptotically unbiased Bayesian inference in a new expressive class of piecewise graphical models and empirically requires orders of magnitude less time than rejection, Metropolis-Hastings, and conventional Gibbs sampling methods to achieve the same level of accuracy.

A blocked Gibbs sampler for NGG-mixture models via a priori truncation

We define a new class of random probability measures, approximating the well-known normalized generalized gamma (NGG) process. Our new process is defined from the representation of NGG processes as discrete measures where the weights are obtained by normalization of the jumps of Poisson processes and the support consists of independent identically distributed location points, however considering only jumps larger than a threshold $$\varepsilon $$?. Therefore, the number of jumps of the new process, called $$\varepsilon $$?-NGG process, is a.s. finite. A prior distribution for $$\varepsilon $$? can be elicited. We assume such a process as the mixing measure in a mixture model for density and cluster estimation, and build an efficient Gibbs sampler scheme to simulate from the posterior. Finally, we discuss applications and performance of the model to two popular datasets, as well as comparison with competitor algorithms, the slice sampler and a posteriori truncation.

Computational Semantics with Functional Programming, by Jan van Eijck and Christina Unger .

One of the fundamental tasks of science is to find explainable relationships between observed phenomena. One approach to this task that has received attention in recent years is based on probabilistic graphical modelling with sparsity constraints on model structures. In this paper, we describe two new approaches to Bayesian inference of sparse structures of Gaussian graphical models (GGMs). One is based on a simple modification of the cutting-edge block Gibbs sampler for sparse GGMs, which results in significant computational gains in high dimensions. The other method is based on a specific construction of the Hamiltonian Monte Carlo sampler, which results in further significant improvements. We compare our fully Bayesian approaches with the popular regularisation-based graphical LASSO, and demonstrate significant advantages of the Bayesian treatment under the same computing costs. We apply the methods to a broad range of simulated data sets, and a real-life financial data set.

Estimating Multiple Pathways of Object Growth Using Nonlongitudinal Image Data

This article presents an infinite Bayesian mixture of monotonic regression models for analyzing multiple growth pathways of star-shaped objects growing over time by using nonlongitudinal data. A motivating example is the analysis of nanocrystal growth processes. A radius function representation used for the outlines of star-shaped objects allows us to represent an object growth (i.e., expansion of outlines) by an increasing sequence of random radius functions. We propose a monotonic regression model to fit the increasing sequence to nonlongitudinal data ensuring that the fitted radius functions always monotonically increase in the sequence. To model the multiple pathways of object growth, we propose the Dirichlet infinite location mixture of multiple monotonic regression models and use a block Gibbs sampler as a numerical solver. We apply the proposed model to nonlongitudinal sets of microscopic image data for inferring multiple growth pathways of nanocrystals. Comparing the inference result with real nanocrystal growth trajectories, we conclude that the growth pathways inferred from the nonlongitudinal data are consistent with the real nanocrystal growth trajectories. Our implementation of the proposed method in Matlab and example datasets are available online as supplementary materials.

Sequential beat-to-beat P and T wave delineation and waveform estimation in ECG signals: Block Gibbs sampler and marginalized particle filter

For ECG interpretation, the detection and delineation of P and T waves are challenging tasks. This paper proposes sequential Bayesian methods for simultaneous detection, threshold-free delineation, and waveform estimation of P and T waves on a beat-to-beat basis. By contrast to state-of-the-art methods that process multiple-beat signal blocks, the proposed Bayesian methods account for beat-to-beat waveform variations by sequentially estimating the waveforms for each beat. Our methods are based on Bayesian signal models that take into account previous beats as prior information. To estimate the unknown parameters of these Bayesian models, we first propose a block Gibbs sampler that exhibits fast convergence in spite of the strong local dependencies in the ECG signal. Then, in order to take into account all the information contained in the past rather than considering only one previous beat, a sequential Monte Carlo method is presented, with a marginalized particle filter that efficiently estimates the unknown parameters of the dynamic model. Both methods are evaluated on the annotated QT database and observed to achieve significant improvements in detection rate and delineation accuracy compared to state-of-the-art methods, thus providing promising approaches for sequential P and T wave analysis.

Nonparametric Bayesian inference for multivariate density functions using Feller priors

Multivariate density estimation plays an important role in investigating the mechanism of high-dimensional data. This article describes a nonparametric Bayesian approach to the estimation of multivariate densities. A general procedure is proposed for constructing Feller priors for multivariate densities and their theoretical properties as nonparametric priors are established. A blocked Gibbs sampling algorithm is devised to sample from the posterior of the multivariate density. A simulation study is conducted to evaluate the performance of the procedure.

Linking Multi-Category Purchases to Latent Activities of Shoppers: Analysing Market Baskets by Topic Models

We investigate the application of two topic models, latent Dirichlet allocation (LDA) and the correlated topic model (CTM), to market basket analysis. Topic models measure the association between observed purchases and underlying latent activities of shoppers by conceiving each basket as random mixture of latent activities. We explain the structure of the two topic models used. We discuss estimation of LDA models by blocked Gibbs sampling. In addition we show how to evaluate the performance of topic models on estimation and holdout data. In the empirical study we analyse a total of 18,000 purchases made at a medium-sized supermarket which refer to 60 product categories. The LDA model performs better than the CTM in terms of log likelihood values. Latent activities inferred by this models are intuitive and interpretable, e.g., related to shopping of beverages or personal care, to baking or to an inclination towards luxury food. To illustrate the managerial relevance of estimated topic models we sketch the core of a recommender system which ranks purchase probabilities of other product categories conditional on the basket of a shopper.

Non-Parametric Bayesian Modelling of Digital Gene Expression Data

Next-generation sequencing technologies provide a revolutionary tool for generating gene expression data. Starting with a fixed RNA sample, they construct a library of millions of differentially abundant short sequence tags or "reads", which constitute a fundamentally discrete measure of the level of gene expression. A common limitation in experiments using these technologies is the low number or even absence of biological replicates, which complicates the statistical analysis of digital gene expression data. Analysis of this type of data has often been based on modified tests originally devised for analysing microarrays; both these and even de novo methods for the analysis of RNA-seq data are plagued by the common problem of low replication. We propose a novel, non-parametric Bayesian approach for the analysis of digital gene expression data. We begin with a hierarchical model for modelling over-dispersed count data and a blocked Gibbs sampling algorithm for inferring the posterior distribution of model parameters conditional on these counts. The algorithm compensates for the problem of low numbers of biological replicates by clustering together genes with tag counts that are likely sampled from a common distribution and using this augmented sample for estimating the parameters of this distribution. The number of clusters is not decided a priori, but it is inferred along with the remaining model parameters. We demonstrate the ability of this approach to model biological data with high fidelity by applying the algorithm on a public dataset obtained from cancerous and non-cancerous neural tissues.

Geometric ergodicity of Gibbs samplers for Bayesian general linear mixed models with proper priors

When a Bayesian version of the general linear mixed model is created by adopting a conditionally conjugate prior distribution, a simple block Gibbs sampler can be employed to explore the resulting intractable posterior density. In this article it is shown that, under mild conditions that nearly always hold in practice, the block Gibbs Markov chain is geometrically ergodic.

Semiparametric Bayes local additive models for longitudinal data.

In longitudinal data analysis, there is great interest in assessing the impact of predictors on the time-varying trajectory in a response variable. In such settings, an important issue is to account for heterogeneity in the shape of the trajectory among subjects, while allowing the impact of the predictors to vary across subjects. We propose a flexible semiparametric Bayes approach for addressing this issue relying on a local partition process prior, which allows flexible local borrowing of information across subjects. Local hypothesis testing and credible bands are developed for the identification of time windows across which a predictor has a significant impact, while adjusting for multiple comparisons. Posterior computation proceeds via an efficient MCMC algorithm using the exact block Gibbs sampler. The methods are assessed using simulation studies and applied to a yeast cell-cycle gene expression data set.

A Bayesian semiparametric dynamic two-level structural equation model for analyzing non-normal longitudinal data

Analyses of non-normal data and longitudinal data to study changes in variables measured repeatedly over time have received considerable attention in social and psychological research. This paper proposes a dynamic two-level nonlinear structural equation model with covariates for analyzing multivariate longitudinal responses that are mixed continuous and ordered categorical variables. To cope with the non-normal continuous data, the corresponding residual errors at both first-level and second-level models are modeled through a Bayesian semiparametric modeling on the basis of a truncated and centered Dirichlet process with stick-breaking priors. The first-level model is defined for measures taken at each time point nested within individuals for investigating their characteristics that vary with time; while the second level is defined for individuals to assess their characteristics that are invariant with time. An algorithm based on the blocked Gibbs sampler is implemented for estimation of parameters. An efficient model comparison statistic, namely the Lν-measure, is also introduced. Results of a simulation study indicate that the performance of the Bayesian semiparametric estimation is satisfactory. The proposed methodologies are applied to a real longitudinal study concerning cocaine use.

Employing MCMC under the PPL framework to analyze sequence data in large pedigrees

The increased feasibility of whole-genome (or whole-exome) sequencing has led to renewed interest in using family data to find disease mutations. For clinical phenotypes that lend themselves to study in large families, this approach can be particularly effective, because it may be possible to obtain strong evidence of a causal mutation segregating in a single pedigree even under conditions of extreme locus and/or allelic heterogeneity at the population level. In this paper, we extend our capacity to carry out positional mapping in large pedigrees, using a combination of linkage analysis and within-pedigree linkage trait-variant disequilibrium analysis to fine map down to the level of individual sequence variants. To do this, we develop a novel hybrid approach to the linkage portion, combining the non-stochastic approach to integration over the trait model implemented in the software package Kelvin, with Markov chain Monte Carlo-based approximation of the marker likelihood using blocked Gibbs sampling as implemented in the McSample program in the JPSGCS package. We illustrate both the positional mapping template, as well as the efficacy of the hybrid algorithm, in application to a single large pedigree with phenotypes simulated under a two-locus trait model.