Bayesian Regression Trees Research Articles

Modeling joint health effects of temperature and air pollution mixtures using Bayesian regression tree ensembles

Modeling joint health effects of temperature and air pollution mixtures using Bayesian regression tree ensembles

Sparse prediction informed by genetic annotations using the logit normal prior for Bayesian regression tree ensembles.

Using high-dimensional genetic variants such as single nucleotide polymorphisms (SNP) to predict complex diseases and traits has important applications in basic research and other clinical settings. For example, predicting gene expression is a necessary first step to identify (putative) causal genes in transcriptome-wide association studies. Due to weak signals, high-dimensionality, and linkage disequilibrium (correlation) among SNPs, building such a prediction model is challenging. However, functional annotations at the SNP level (e.g., as epigenomic data across multiple cell- or tissue-types) are available and could be used to inform predictor importance and aid in outcome prediction. Existing approaches to incorporate annotations have been based mainly on (generalized) linear models. Bayesian additive regression trees (BART), in contrast, is a reliable method to obtain high-quality nonlinear out of sample predictions without overfitting. Unfortunately, the default prior from BART may be too inflexible to handle sparse situations where the number of predictors approaches or surpasses the number of observations. Motivated by our real data application, this article proposes an alternative prior based on the logit normal distribution because it provides a framework that is adaptive to sparsity and can model informative functional annotations. It also provides a framework to incorporate prior information about the between SNP correlations. Computational details for carrying out inference are presented along with the results from a simulation study and a genome-wide prediction analysis of the Alzheimer's Disease Neuroimaging Initiative data.

The taxicab sampler: MCMC for discrete spaces with application to tree models

Motivated by the problem of exploring discrete but very complex state spaces in Bayesian models, we propose a novel Markov Chain Monte Carlo search algorithm: the taxicab sampler. We describe the construction of this sampler and discuss how its interpretation and usage differs from that of standard Metropolis-Hastings as well as the related Hamming ball sampler. The proposed sampling algorithm is then shown to demonstrate substantial improvement in computation time without any loss of efficiency relative to a naïve Metropolis–Hastings search in a motivating Bayesian regression tree count model, in which we leverage the discrete state space assumption to construct a novel likelihood function that allows for flexibly describing different mean-variance relationships while preserving parameter interpretability compared to existing likelihood functions for count data.

Estimating perinatal critical windows of susceptibility to environmental mixtures via structured Bayesian regression tree pairs.

Maternal exposure to environmental chemicals during pregnancy can alter birth and children's health outcomes. Research seeks to identify critical windows, time periods when exposures can change future health outcomes, and estimate the exposure-response relationship. Existing statistical approaches focus on estimation of the association between maternal exposure to a single environmental chemical observed at high temporal resolution (e.g., weekly throughout pregnancy) and children's health outcomes. Extending to multiple chemicals observed at high temporal resolution poses a dimensionality problem and statistical methods are lacking. We propose a regression tree-based model for mixtures of exposures observed at high temporal resolution. The proposed approach uses an additive ensemble of tree pairs that defines structured main effects and interactions between time-resolved predictors and performs variable selection to select out of the model predictors not correlated with the outcome. In simulation, we show that the tree-based approach performs better than existing methods for a single exposure and can accurately estimate critical windows in the exposure-response relation for mixtures. We apply our method to estimate the relationship between five exposures measured weekly throughout pregnancy and birth weight in a Denver, Colorado, birth cohort. We identified critical windows during which fine particulate matter, sulfur dioxide, and temperature are negatively associated with birth weight and an interaction between fine particulate matter and temperature. Software is made available in the R package dlmtree.

Estimating Perinatal Critical Windows to Environmental Mixtures via Structured Bayesian Regression Tree Pairs

BACKGROUND AND AIM: Maternal exposure to environmental chemicals during pregnancy can alter birth and children’s health outcomes. Research seeks to identify critical windows, time periods when exposures can change future health outcomes, and estimate the exposure-response relationship. Existing statistical approaches that allow for high temporal resolution (e.g. weekly throughout pregnancy) exposure measurements primarily account for a single environmental chemical. Extending to multiple chemicals observed at high temporal resolution poses a dimensionality problem and statistical methods are lacking. METHODS: We propose a regression tree-based model for mixtures of exposures observed at high temporal resolution. The proposed approach uses an additive ensemble of tree pairs that define structured main effects and interactions between time-resolved predictors and performs variable selection. In simulation, we show the tree-based approach performs better than existing methods for a single exposure and can accurately estimate critical windows in the exposure-response relation for mixtures. Software is made available in the R package dlmtree. RESULTS:We apply our method to estimate the relationship between five exposures measured weekly throughout pregnancy and resulting birth weight in a Denver, Colorado birth cohort. We identified critical windows during which fine particulate matter (PM₂₅), sulfur dioxide, and temperature are negatively associated with birth weight and an interaction between PM₂₅ and temperature. CONCLUSIONS:In this work we proposed a method to estimate the association between mixtures of environmental exposures observed at high-temporal resolution on birth and children’s health outcomes. Our analysis is the first analysis to identify critical windows to a mixture observed at high-temporal resolution within a distributed lag mixture model framework. As the size and resolution of the exposure data available continues to grow, our method fills a critical research gap in statistical methods for epidemiology and environmental health in being able to estimate the effects of time-resolved measures of a mixture on a continuous or binary health outcome. KEYWORDS: air pollution, Bayesian additive regression trees, birth outcomes, critical windows, distributed lag models

Assessing variable activity for Bayesian regression trees

Bayesian Additive Regression Trees (BART) are non-parametric models that can capture complex exogenous variable effects. In any regression problem, it is often of interest to learn which variables are most active. Variable activity in BART is usually measured by counting the number of times a tree splits for each variable. Such one-way counts have the advantage of fast computations. Despite their convenience, one-way counts have several issues. They are statistically unjustified, cannot distinguish between main effects and interaction effects, and become inflated when measuring interaction effects. An alternative method well-established in the literature is Soboĺ indices, a variance-based global sensitivity analysis technique. However, these indices often require Monte Carlo integration, which can be computationally expensive. This paper provides analytic expressions for Soboĺ indices for BART posterior samples. These expressions are easy to interpret and are computationally feasible. Furthermore, we will show a fascinating connection between first-order (main-effects) Soboĺ indices and one-way counts. We also introduce a novel ranking method, and use this to demonstrate that the proposed indices preserve the Soboĺ-based rank order of variable importance. Finally, we compare these methods using analytic test functions and the En-ROADS climate impacts simulator.

Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects (with Discussion)

Hahn et al. (2020) offers an extensive study to explicate and evaluate the performance of the BCF model in different settings and provides a detailed discussion about its utility in causal inference. It is a welcomed addition to the causal machine learning literature. I will emphasize the contribution of the BCF model to the field of causal inference through discussions on two topics: 1) the difference between the PS in the BCF model and the Bayesian PS in a Bayesian updating approach, 2) an alternative exposition of the role of the PS in outcome modeling based methods for the estimation of causal effects. I will conclude with comments on avenues for future research involving BCF that will be important and much needed in the era of Big data.

Posterior concentration for Bayesian regression trees and forests

Since their inception in the 1980s, regression trees have been one of the more widely used nonparametric prediction methods. Tree-structured methods yield a histogram reconstruction of the regression surface, where the bins correspond to terminal nodes of recursive partitioning. Trees are powerful, yet susceptible to overfitting. Strategies against overfitting have traditionally relied on pruning greedily grown trees. The Bayesian framework offers an alternative remedy against overfitting through priors. Roughly speaking, a good prior charges smaller trees where overfitting does not occur. While the consistency of random histograms, trees and their ensembles has been studied quite extensively, the theoretical understanding of the Bayesian counterparts has been missing. In this paper, we take a step toward understanding why/when do Bayesian trees and forests not overfit. To address this question, we study the speed at which the posterior concentrates around the true smooth regression function. We propose a spike-and-tree variant of the popular Bayesian CART prior and establish new theoretical results showing that regression trees (and forests) (a) are capable of recovering smooth regression surfaces (with smoothness not exceeding one), achieving optimal rates up to a log factor, (b) can adapt to the unknown level of smoothness and (c) can perform effective dimension reduction when $p>n$. These results provide a piece of missing theoretical evidence explaining why Bayesian trees (and additive variants thereof?) have worked so well in practice.

Assessing the Russian Internet Research Agency’s impact on the political attitudes and behaviors of American Twitter users in late 2017

There is widespread concern that Russia and other countries have launched social-media campaigns designed to increase political divisions in the United States. Though a growing number of studies analyze the strategy of such campaigns, it is not yet known how these efforts shaped the political attitudes and behaviors of Americans. We study this question using longitudinal data that describe the attitudes and online behaviors of 1,239 Republican and Democratic Twitter users from late 2017 merged with nonpublic data about the Russian Internet Research Agency (IRA) from Twitter. Using Bayesian regression tree models, we find no evidence that interaction with IRA accounts substantially impacted 6 distinctive measures of political attitudes and behaviors over a 1-mo period. We also find that interaction with IRA accounts were most common among respondents with strong ideological homophily within their Twitter network, high interest in politics, and high frequency of Twitter usage. Together, these findings suggest that Russian trolls might have failed to sow discord because they mostly interacted with those who were already highly polarized. We conclude by discussing several important limitations of our study-especially our inability to determine whether IRA accounts influenced the 2016 presidential election-as well as its implications for future research on social media influence campaigns, political polarization, and computational social science.

PNS212 AUTOMATING VARIABLE SELECTION IN META-REGRESSION

Meta-regression is used to adjust for sources of treatment heterogeneity in meta-analysis, but are often underpowered. Current best practice for identifying candidates for meta-regression is to use effect modifiers identified in sub-group analysis but these are rare and typically reviewers are forced to rely on clinical opinion. The increased accessibility of statistical models designed to detect sparse signals may offer a viable alternative. We conducted a simulation study comparing two simulated clinical experts to a Bayesian analysis with a sparsity inducing horseshoe prior, and a weighted random forest implemented in the R package metaforest. clinician one used a rule of one covariate for every 10 studies, while clinician two included all variables believed to be important. Five hundred simulations of 20, 40, and 60 studies were conducted using an outcome model including two true and eight noise variables. Success was evaluated as the ratio of means (RoM), coverage, and confidence interval width. Both clinician approaches performed similarly in terms of RoM (20, 40, 60 studies: 1.27/1.32, 1.32/1.37, 1.29/1.32) although confidence intervals (CIs) for clinician one were always the largest. Metaforest resulted in the lowest bias (RoM = 1.23, 1.07, 1.03) and second CIs while ranking highest for coverage when 40 or more studies were included. The Bayesian horseshoe aggressively shrunk coefficients to zero and had universally poor performance (RoM = 2.20, 1.71, 1.43) but the smallest CIs. The metaforest package offers a fast and accessible interface to detect and visualize effect modifiers and may be a valuable addition to meta-analysis workflow as a compliment to existing theoretically driven approaches. Future work should investigate hybrid approaches (e.g. using clinical expertise) to flag promising coefficients in horseshoe or Bayesian regression trees.

Bayesian Regression Tree Ensembles that Adapt to Smoothness and Sparsity

SummaryEnsembles of decision trees are a useful tool for obtaining flexible estimates of regression functions. Examples of these methods include gradient-boosted decision trees, random forests and Bayesian classification and regression trees. Two potential shortcomings of tree ensembles are their lack of smoothness and their vulnerability to the curse of dimensionality. We show that these issues can be overcome by instead considering sparsity inducing soft decision trees in which the decisions are treated as probabilistic. We implement this in the context of the Bayesian additive regression trees framework and illustrate its promising performance through testing on benchmark data sets. We provide strong theoretical support for our methodology by showing that the posterior distribution concentrates at the minimax rate (up to a logarithmic factor) for sparse functions and functions with additive structures in the high dimensional regime where the dimensionality of the covariate space is allowed to grow nearly exponentially in the sample size. Our method also adapts to the unknown smoothness and sparsity levels, and can be implemented by making minimal modifications to existing Bayesian additive regression tree algorithms.

Bayesian Regression Trees for High-Dimensional Prediction and Variable Selection

ABSTRACTDecision tree ensembles are an extremely popular tool for obtaining high-quality predictions in nonparametric regression problems. Unmodified, however, many commonly used decision tree ensemble methods do not adapt to sparsity in the regime in which the number of predictors is larger than the number of observations. A recent stream of research concerns the construction of decision tree ensembles that are motivated by a generative probabilistic model, the most influential method being the Bayesian additive regression trees (BART) framework. In this article, we take a Bayesian point of view on this problem and show how to construct priors on decision tree ensembles that are capable of adapting to sparsity in the predictors by placing a sparsity-inducing Dirichlet hyperprior on the splitting proportions of the regression tree prior. We characterize the asymptotic distribution of the number of predictors included in the model and show how this prior can be easily incorporated into existing Markov chain Monte Carlo schemes. We demonstrate that our approach yields useful posterior inclusion probabilities for each predictor and illustrate the usefulness of our approach relative to other decision tree ensemble approaches on both simulated and real datasets. Supplementary materials for this article are available online.

Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects

This paper presents a novel nonlinear regression model for estimating heterogeneous treatment effects from observational data, geared specifically towards situations with small effect sizes, heterogeneous effects, and strong confounding. Standard nonlinear regression models, which may work quite well for prediction, have two notable weaknesses when used to estimate heterogeneous treatment effects. First, they can yield badly biased estimates of treatment effects when fit to data with strong confounding. The Bayesian causal forest model presented in this paper avoids this problem by directly incorporating an estimate of the propensity function in the specification of the response model, implicitly inducing a covariate-dependent prior on the regression function. Second, standard approaches to response surface modeling do not provide adequate control over the strength of regularization over effect heterogeneity. The Bayesian causal forest model permits treatment effect heterogeneity to be regularized separately from the prognostic effect of control variables, making it possible to informatively shrink to homogeneity. We illustrate these benefits via the reanalysis of an observational study assessing the causal effects of smoking on medical expenditures as well as extensive simulation studies.

Efficient Metropolis–Hastings Proposal Mechanisms for Bayesian Regression Tree Models

Bayesian regression trees are flexible non-parametric models that are well suited to many modern statistical regression problems. Many such tree models have been proposed, from the simple single- tree model to more complex tree ensembles. Their non-parametric formulation allows for effective and efficient modeling of datasets exhibiting complex non-linear relationships between the model pre- dictors and observations. However, the mixing behavior of the Markov Chain Monte Carlo (MCMC) sampler is sometimes poor. This is because the proposals in the sampler are typically local alterations of the tree structure, such as the birth/death of leaf nodes, which does not allow for efficient traversal of the model space. This poor mixing can lead to inferential problems, such as under-representing uncertainty. In this paper, we develop novel proposal mechanisms for efficient sampling. The first is a rule perturbation proposal while the second we call tree rotation. The perturbation proposal can be seen as an efficient variation of the change proposal found in existing literature. The novel tree rotation proposal is simple to implement as it only requires local changes to the regression tree structure, yet it efficiently traverses disparate regions of the model space along contours of equal probability. When combined with the classical birth/death proposal, the resulting MCMC sampler exhibits good acceptance rates and properly represents model uncertainty in the posterior samples. We implement this sampling algorithm in the Bayesian Additive Regression Tree (BART) model and demonstrate its effectiveness on a prediction problem from computer experiments and a test function where structural tree variability is needed to fully explore the posterior.

Contributed Discussion on Article by Pratola

The author should be commended for his outstanding contribution to the literature on Bayesian regression tree models. The author introduces three innovative sampling approaches which allow for efficient traversal of the model space. In this response, we add a fourth alternative.

Application of Propensity Scores to a Continuous Exposure: Effect of Lead Exposure in Early Childhood on Reading and Mathematics Scores

The estimation of causal effects in observational studies is usually limited by the lack of randomization, which can result in different treatment or exposure groups differing systematically with respect to characteristics that influence outcomes. To remove such systematic differences, methods to ”balance” subjects on observed covariates across treatment or exposure levels have been developed over the past three decades. These methods have been primarily developed in settings with binary treatment or exposures. However, in many observational studies, the exposures are continuous instead of being binary or discrete, and are usually considered as doses of treatment. In this manuscript we consider estimating the causal effect of early childhood lead exposure on youth academic achievement, where the exposure variable blood lead concentration can take any values that are greater than or equal to 0, using three balancing methods: propensity score analysis, non-bipartite matching, and Bayesian regression trees. We find some evidence that the standard logistic regression analysis controlling for age and socioeconomic confounders used in previous analyses (Zhang et al. (2013)) overstates the effect of lead exposure on performance on standardized mathematics and reading examinations; however, significant declines remain, including at doses currently below the recommended exposure levels.

A Bayesian regression tree approach to identify the effect of nanoparticles’ properties on toxicity profiles

We introduce a Bayesian multiple regression tree model to characterize relationships between physico-chemical properties of nanoparticles and their in-vitro toxicity over multiple doses and times of exposure. Unlike conventional models that rely on data summaries, our model solves the low sample size issue and avoids arbitrary loss of information by combining all measurements from a general exposure experiment across doses, times of exposure, and replicates. The proposed technique integrates Bayesian trees for modeling threshold effects and interactions, and penalized B-splines for dose- and time-response surface smoothing. The resulting posterior distribution is sampled by Markov Chain Monte Carlo. This method allows for inference on a number of quantities of potential interest to substantive nanotoxicology, such as the importance of physico-chemical properties and their marginal effect on toxicity. We illustrate the application of our method to the analysis of a library of 24 nano metal oxides.