Bayesian Nonparametric Regression Research Articles

Nonparametric survival regression using the beta-Stacy process

A novel class of hierarchical nonparametric Bayesian survival regression models for time-to-event data with uninformative right censoring is introduced. The survival curve is modeled as a random function whose prior distribution is defined using the beta-Stacy (BS) process. The prior mean of each survival probability and its prior variance are linked to a standard parametric survival regression model. This nonparametric survival regression can thus be anchored to any reference parametric form, such as a proportional hazards or an accelerated failure time model, allowing substantial departures of the predictive survival probabilities when the reference model is not supported by the data. Also, under this formulation the predictive survival probabilities will be close to the empirical survival distribution near the mode of the reference model and they will be shrunken towards its probability density in the tails of the empirical distribution.Examples of our approach are presented using a Weibull-centered BS process whose variance decreases proportionally to the centering Weibull probability density function. By the conjugacy of the BS process to random right censoring, marginal posterior inferences for the survival probabilities and for the regression coefficients are approximated using the standard Gibbs sampler. Three applications of the BS Weibull survival regression model are illustrated, focussing on toxicological and clinical data and comparing the BS model estimates with those of standard parametric and semi-parametric survival regressions.

Bayesian Nonparametric Shrinkage Applied to Cepheid Star Oscillations

Bayesian nonparametric regression with dependent wavelets has dual shrinkage properties: there is shrinkage through a dependent prior put on functional differences, and shrinkage through the setting of most of the wavelet coefficients to zero through Bayesian variable selection methods. The methodology can deal with unequally spaced data and is efficient because of the existence of fast moves in model space for the MCMC computation. The methodology is illustrated on the problem of modeling the oscillations of Cepheid variable stars; these are a class of pulsating variable stars with the useful property that their periods of variability are strongly correlated with their absolute luminosity. Once this relationship has been calibrated, knowledge of the period gives knowledge of the luminosity. This makes these stars useful as "standard candles" for estimating distances in the universe.

Immunologic profile of excessive body weight

The purpose of this paper is to identify immunologic hallmarks of excessive bodyweight. The analysis is based on 176 adults (106 women, 70 men) who participated in a nested case-control study in Italy. All participants were healthy at the time of blood collection and aged between 36 and 75 years. We employed multivariate analysis of variance and a nonparametric Bayesian additive regression tree approach along with a receiver operating characteristic (ROC) curve analysis to determine the immunologic signature of excessive body weight (i.e., obesity and overweight). Interleukin 8 (IL-8), IL-10, interferon γ, and inducible protein 10 were shown to be predictive of excessive body weight with an area under the ROC curve of 71% (p < 0.0002). We propose that by using this profile-based approach to define immunologic signatures, it might be possible to identify unique immunologic hallmarks of specific types of obesity.

Optimal U-type design for Bayesian nonparametric multiresponse prediction

This paper presents an extension of the work of Yue and Chatterjee (2010) about U-type designs for Bayesian nonparametric response prediction. We consider nonparametric Bayesian regression model with p responses. We use U-type designs with n runs, m factors and q levels for the nonparametric multiresponse prediction based on the asymptotic Bayesian criterion. A lower bound for the proposed criterion is established, and some optimal and nearly optimal designs for the illustrative models are given.

Priors for Bayesian adaptive spline smoothing

Adaptive smoothing has been proposed for curve-fitting problems where the underlying function is spatially inhomogeneous. Two Bayesian adaptive smoothing models, Bayesian adaptive smoothing splines on a lattice and Bayesian adaptive P-splines, are studied in this paper. Estimation is fully Bayesian and carried out by efficient Gibbs sampling. Choice of prior is critical in any Bayesian non-parametric regression method. We use objective priors on the first level parameters where feasible, specifically independent Jeffreys priors (right Haar priors) on the implied base linear model and error variance, and we derive sufficient conditions on higher level components to ensure that the posterior is proper. Through simulation, we demonstrate that the common practice of approximating improper priors by proper but diffuse priors may lead to invalid inference, and we show how appropriate choices of proper but only weakly informative priors yields satisfactory inference.

Using Multivariate Regression Methods to Resolve Overlapped Electrochemical Signals

This paper proposes the application of Gaussian process regression (GPR) as an alternative regression model to resolve the hard overlapped electrochemical signals belonging to the 2,4,6trichlorophenol/2,6-dichlorophenol (TCP/DCP) system. Gaussian process derives from the perspective of Bayesian non-parametric regression methods, in terms of the parameterization of the covariance function, results in its good performance for the development of a calibration model for both linear and non-linear data sets. The multivariate regression model developed by GPR was compared with some traditional regression methods such as partial least squares regression (PLSR), and support vector regression (SVR). The comparative results were satisfied. The satisfactory results obtained through GPR method suggest that it can be used as a more effective and promising tool for multivariate regression tasks than the others.

Bayesian Nonparametric Regression Analysis of Data with Random Effects Covariates from Longitudinal Measurements

We consider nonparametric regression analysis in a generalized linear model (GLM) framework for data with covariates that are the subject-specific random effects of longitudinal measurements. The usual assumption that the effects of the longitudinal covariate processes are linear in the GLM may be unrealistic and if this happens it can cast doubt on the inference of observed covariate effects. Allowing the regression functions to be unknown, we propose to apply Bayesian nonparametric methods including cubic smoothing splines or P-splines for the possible nonlinearity and use an additive model in this complex setting. To improve computational efficiency, we propose the use of data-augmentation schemes. The approach allows flexible covariance structures for the random effects and within-subject measurement errors of the longitudinal processes. The posterior model space is explored through a Markov chain Monte Carlo (MCMC) sampler. The proposed methods are illustrated and compared to other approaches, the "naive" approach and the regression calibration, via simulations and by an application that investigates the relationship between obesity in adulthood and childhood growth curves.

BART: Bayesian additive regression trees

We develop a Bayesian "sum-of-trees" model where each tree is constrained by a regularization prior to be a weak learner, and fitting and inference are accomplished via an iterative Bayesian backfitting MCMC algorithm that generates samples from a posterior. Effectively, BART is a nonparametric Bayesian regression approach which uses dimensionally adaptive random basis elements. Motivated by ensemble methods in general, and boosting algorithms in particular, BART is defined by a statistical model: a prior and a likelihood. This approach enables full posterior inference including point and interval estimates of the unknown regression function as well as the marginal effects of potential predictors. By keeping track of predictor inclusion frequencies, BART can also be used for model-free variable selection. BART's many features are illustrated with a bake-off against competing methods on 42 different data sets, with a simulation experiment and on a drug discovery classification problem.

Variable Selection in Bayesian Smoothing Spline ANOVA Models: Application to Deterministic Computer Codes

With many predictors, choosing an appropriate subset of the covariates is a crucial—and difficult—step in nonparametric regression. We propose a Bayesian nonparametric regression model for curve fitting and variable selection. We use the smoothing splines ANOVA framework to decompose the regression function into interpretable main effect and interaction functions, and use stochastic search variable selection through Markov chain Monte Carlo sampling to search for models that fit the data well. We also show that variable selection is highly sensitive to hyperparameter choice, and develop a technique for selecting hyperparameters that control the long-run false-positive rate. We use our method to build an emulator for a complex computer model for two-phase fluid flow.

A nonparametric dependent process for Bayesian regression

This article investigates the problem of Bayesian nonparametric regression. The proposed model is based on a recently introduced random distribution function, which is based on a decreasing set of weights. The approach is surprisingly of a much simpler form than alternative models described in the literature. A Gibbs sampler algorithm is provided for posterior analysis.

Simultaneous inference for multiple testing and clustering via a Dirichlet process mixture model

We propose a Bayesian nonparametric regression model that exploits clustering for increased sensitivity in multiple hypothesis testing. We build on the recently proposed BEMMA (Bayesian Effects Models for Microarrays) method which is able to model dependence among objects through clustering and then estimates hypothesis-testing parameters averaged over clustering uncertainty. We propose several improvements. First, we separate the clustering of the regression coefficients from the part of the model that accommodates heteroscedasticity. Second, our model accommodates a wider class of experimental designs, such as permitting covariates and not requiring independent sampling. Third, we provide a more satisfactory treatment of nuisance parameters and some hyperparameters. Finally, we do not require the arbitrary designation of a reference treatment. The proposed method is compared in a simulation study to ANOVA and the BEMMA methods.

O(N 2)-Operation Approximation of Covariance Matrix Inverse in Gaussian Process Regression Based on Quasi-Newton BFGS Method

Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. However, during its model-tuning procedure, the GP implementation suffers from numerous covariance-matrix inversions of expensive O(N3) operations, where N is the matrix dimension. In this article, we propose using the quasi-Newton BFGS O(N2)-operation formula to approximate/replace recursively the inverse of covariance matrix at every iteration. The implementation accuracy is guaranteed carefully by a matrix-trace criterion and by the restarts technique to generate good initial guesses. A number of numerical tests are then performed based on the sinusoidal regression example and the Wiener–Hammerstein identification example. It is shown that by using the proposed implementation, more than 80% O(N3) operations could be eliminated, and a typical speedup of 5–9 could be achieved as compared to the standard maximum-likelihood-estimation (MLE) implementation commonly used in Gaussian process regression.

Gaussian process regression for multivariate spectroscopic calibration

Traditionally multivariate calibration models have been developed using regression based techniques including principal component regression and partial least squares and their non-linear counterparts. This paper proposes the application of Gaussian process regression as an alternative method for the development of a calibration model. By formulating the regression problem in a probabilistic framework, a Gaussian process is derived from the perspective of Bayesian non-parametric regression, prior to describing its implementation using Markov chain Monte Carlo methods. The flexibility of a Gaussian process, in terms of the parameterization of the covariance function, results in its good performance in terms of the development of a calibration model for both linear and non-linear data sets. To handle the high dimensionality of spectral data, principal component analysis is initially performed on the data, followed by the application of Gaussian process regression to the scores of the extracted principal components. In this sense, the proposed method is a non-linear variant of principal component regression. The effectiveness of the Gaussian process approach for the development of a calibration model is demonstrated through its application to two spectroscopic data sets. A statistical hypothesis test procedure, the paired t-test, is used to undertake an empirical comparison of the Gaussian process approach with conventional calibration techniques, and it is concluded that the Gaussian process exhibits enhanced behaviour.

Misinformation in the conjugate prior for the linear model with implications for free-knot spline modelling

In the conjugate prior for the normal linear model, the prior variance for the coefficients is a multiple of the error variance parameter. However, if the prior mean for the coefficients is poorly chosen, the posterior distribution of the model can be seriously distorted because of prior dependence between the coefficients and error variance. In particular, the error variance will be overestimated, as will the posterior variance of the coefficients. This occurs because the prior mean, which can be thought of as a weighted pseudo-observation, is an outlier with respect to the real observations. While this situation will be easily noticed and avoided in simple models, in more complicated models, the effect can be easily overlooked. The issue arises in the unit information (UI) prior, a conjugate prior in which the prior contributes information equal to that in one observation. In particular, a successful Bayesian nonparametric regression model - Bayesian Adaptive Regression Splines (BARS) - that relies on the UI prior for its model selection step suffers from this problem, and addressing the problem within the Bayesian paradigm alters the penalty on model dimensionality.

Exploiting Hessian matrix and trust-region algorithm in hyperparameters estimation of Gaussian process

Gaussian process (GP) regression is a Bayesian non-parametric regression model, showing good performance in various applications. However, it is quite rare to see research results on log-likelihood maximization algorithms. Instead of the commonly used conjugate gradient method, the Hessian matrix is first derived/simplified in this paper and the trust-region optimization method is then presented to estimate GP hyperparameters. Numerical experiments verify the theoretical analysis, showing the advantages of using Hessian matrix and trust-region algorithms. In the GP context, the trust-region optimization method is a robust alternative to conjugate gradient method, also in view of future researches on approximate and/or parallel GP-implementation.

EFFICIENT GAUSSIAN PROCESS BASED ON BFGS UPDATING AND LOGDET APPROXIMATION

Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. However, its hyperparameter-estimation procedure suffers from numerous covariance-matrix inversions of prohibitively O(N3) operations. In this paper, we propose using the quasi-Newton BFGS O(N2)-operation formula to update recursively the inverse of covariance matrix at every iteration. As for the involved log det computation, a power-series expansion based approximation and compensation scheme is proposed with only 50 N2 operations. A number of numerical tests are performed based on the 2D-sinusoidal regression example and the Wiener-Hammerstein identification example. It is shown that by using the proposed implementation, more than 80% O(N3) operations are eliminated, and the speedup of 5 ~9 can be achieved.

U-type and factorial designs for nonparametric Bayesian regression

This paper deals with the design problem for recovering a response surface by using a nonparametric Bayesian approach. The criterion for selecting the designs is based on the asymptotic average estimation variance, and three priors for the response are specified. We found the optimal design that minimizes the criterion over the lattice designs with s q-level factors and N runs. The approach we used is similar to that in Ma et al. (J. Statist. Plann. Inference 113 (2003) 323). We also obtained alternative expressions and lower bounds for the criterion corresponding to each of the three Bayes models for the two-level U-type design by using the column balance and row distance proposed in Fang et al. (J. Complexity 19 (2003) 692). These results mat be used to construct the two-level U-type designs for the nonparametric Bayesian models.

Dynamic Bayesian network and nonparametric regression for nonlinear modeling of gene networks from time series gene expression data

We propose a dynamic Bayesian network and nonparametric regression model for constructing a gene network from time series microarray gene expression data. The proposed method can overcome a shortcoming of the Bayesian network model in the sense of the construction of cyclic regulations. The proposed method can analyze the microarray data as a continuous data and can capture even nonlinear relations among genes. It can be expected that this model will give a deeper insight into complicated biological systems. We also derive a new criterion for evaluating an estimated network from Bayes approach. We conduct Monte Carlo experiments to examine the effectiviness of the proposed method. We also demonstrate the proposed method through the analysis of the Saccharomyces cerevisiae gene expression data.

Gaussian Process Regression for Batch Process Modelling

This paper proposes the application of Gaussian process regression for the empirical modelling of batch processes to provide long range predictions. Gaussian processes are flexible, non-parametric Bayesian regression techniques. In the training stage, hyper-parameters that define the covariance structure of the Gaussian process can be obtained using Markov Chain Monte Carlo sampling. Model predictions can then be achieved by taking the average of the Monte Carlo samples. The proposed technique is evaluated by application to a benchmark simulation of a fed-batch bioreactor. The results show that comparable results can be achieved with other non-parametric modelling approaches such as recurrent neural networks.

Inference of gene regulatory networks by means of dynamic differential Bayesian networks and nonparametric regression.

We propose a dynamic differential Bayesian networks (DDBNs) and nonparametric regression model. This model is an extended model of traditional dynamic Bayesian networks (DBNs), which can incorporate temporal information in a natural way and directly handle real-valued data obtained from microarrays without any transformation. In addition, it can cope with differential information between gene expression levels, without any loss to the traditional advantage, i.e., the capability of estimating non-linear relationships between genes. We apply DDBNs to analyze simulated data and real data, i.e., Saccharomyces cerevisiae cell cycle gene expression data. We have confirmed the effectiveness of our approach in the sense that some edges have been successfully detected only by DDBNs, not by DBNs.