Remaining Useful Life Prediction Using Bayesian Additive Regression Trees

Additive Regression Trees (BART) for Normalized Left Ventricular Mass (LVM), LVM and Left Ventricular Hypertrophy (LVH).https://www.mcw.

There is a dearth of robust methods to estimate the causal effects of multiple treatments when the outcome is binary. This paper uses two unique sets of simulations to propose and evaluate the use of Bayesian additive regression trees in such settings. First, we compare Bayesian additive regression trees to several approaches that have been proposed for continuous outcomes, including inverse probability of treatment weighting, targeted maximum likelihood estimator, vector matching, and regression adjustment. Results suggest that under conditions of non-linearity and non-additivity of both the treatment assignment and outcome generating mechanisms, Bayesian additive regression trees, targeted maximum likelihood estimator, and inverse probability of treatment weighting using generalized boosted models provide better bias reduction and smaller root mean squared error. Bayesian additive regression trees and targeted maximum likelihood estimator provide more consistent 95% confidence interval coverage and better large-sample convergence property. Second, we supply Bayesian additive regression trees with a strategy to identify a common support region for retaining inferential units and for avoiding extrapolating over areas of the covariate space where common support does not exist. Bayesian additive regression trees retain more inferential units than the generalized propensity score-based strategy, and shows lower bias, compared to targeted maximum likelihood estimator or generalized boosted model, in a variety of scenarios differing by the degree of covariate overlap. A case study examining the effects of three surgical approaches for non-small cell lung cancer demonstrates the methods.

BackgroundThe goal of genome-wide prediction (GWP) is to predict phenotypes based on marker genotypes, often obtained through single nucleotide polymorphism (SNP) chips. The major problem with GWP is high-dimensional data from many thousands of SNPs scored on several thousands of individuals. A large number of methods have been developed for GWP, which are mostly parametric methods that assume statistical linearity and only additive genetic effects. The Bayesian additive regression trees (BART) method was recently proposed and is based on the sum of nonparametric regression trees with the priors being used to regularize the parameters. Each regression tree is based on a recursive binary partitioning of the predictor space that approximates an unknown function, which will automatically model nonlinearities within SNPs (dominance) and interactions between SNPs (epistasis). In this study, we introduced BART and compared its predictive performance with that of the LASSO, Bayesian LASSO (BLASSO), genomic best linear unbiased prediction (GBLUP), reproducing kernel Hilbert space (RKHS) regression and random forest (RF) methods.ResultsTests on the QTLMAS2010 simulated data, which are mainly based on additive genetic effects, show that cross-validated optimization of BART provides a smaller prediction error than the RF, BLASSO, GBLUP and RKHS methods, and is almost as accurate as the LASSO method. If dominance and epistasis effects are added to the QTLMAS2010 data, the accuracy of BART relative to the other methods was increased. We also showed that BART can produce importance measures on the SNPs through variable inclusion proportions. In evaluations using real data on pigs, the prediction error was smaller with BART than with the other methods.ConclusionsBART was shown to be an accurate method for GWP, in which the regression trees guarantee a very sparse representation of additive and complex non-additive genetic effects. Moreover, the Markov chain Monte Carlo algorithm with Bayesian back-fitting provides a computationally efficient procedure that is suitable for high-dimensional genomic data.Electronic supplementary materialThe online version of this article (doi:10.1186/s12711-016-0219-8) contains supplementary material, which is available to authorized users.

Bayesian Additive Regression Trees (BART) is a relatively new model within the domain of statistical/machine learning. BART has seen rapid development in recent years, having been extended and adapted to new application areas. Given that BART is a fully Bayesian model, care should be taken to justify that samples drawn via Markov Chain Monte Carlo (MCMC) from BART’s posterior distribution can be regarded as from a stationary posterior distribution. Presently, no formal method exists for conducting such diagnostic checks. As such, a formal convergence criterion is developed for BART called the Posterior Tree Deviance (PTD). This method for assessing convergence of BART’s MCMC sampler is implemented in a novel software package, BART.jl, written in the Julia programming language. Working with BART presents an onerous software burden. BART.jl contains a much smaller codebase than implementations in other programming languages and provides user-friendly functionality for working with BART’s ensemble of decision trees. Lastly, BART is adapted to two novel application areas: (1) variable selection in the presence of mandatory and optional covariates (2) correcting for bias resulting from a predictor measured with error.

Individualized treatment rules can improve health outcomes by recognizing that patients may respond differently to treatment and assigning therapy with the most desirable predicted outcome for each individual. Flexible and efficient prediction models are desired as a basis for such individualized treatment rules to handle potentially complex interactions between patient factors and treatment. Modern Bayesian semiparametric and nonparametric regression models provide an attractive avenue in this regard as these allow natural posterior uncertainty quantification of patient specific treatment decisions as well as the population wide value of the prediction-based individualized treatment rule. In addition, via the use of such models, inference is also available for the value of the optimal individualized treatment rules. We propose such an approach and implement it using Bayesian Additive Regression Trees as this model has been shown to perform well in fitting nonparametric regression functions to continuous and binary responses, even with many covariates. It is also computationally efficient for use in practice. With Bayesian Additive Regression Trees, we investigate a treatment strategy which utilizes individualized predictions of patient outcomes from Bayesian Additive Regression Trees models. Posterior distributions of patient outcomes under each treatment are used to assign the treatment that maximizes the expected posterior utility. We also describe how to approximate such a treatment policy with a clinically interpretable individualized treatment rule, and quantify its expected outcome. The proposed method performs very well in extensive simulation studies in comparison with several existing methods. We illustrate the usage of the proposed method to identify an individualized choice of conditioning regimen for patients undergoing hematopoietic cell transplantation and quantify the value of this method of choice in relation to the optimal individualized treatment rule as well as non-individualized treatment strategies.

Accurate peak load prediction is an important element of the daily planning, operation, and dispatch scheduling in electric utilities. In order to build a better distribution-level peak load prediction reference model for the local utility in Albuquerque, a new nonparametric method, Bayesian Additive Regression Trees (BART), is introduced. First, a detailed analysis of peak load and local weather information from 2012 and 2013 are used. Strong linear relationship is displayed between the peak load and various weather factors during the winter and spring months and nonlinear relationship dominates in the summer and fall months. Next, the BART method is applied with a principled permutation-based inferential variable selection approach. The BART method's prediction accuracy is then compared with the Multiple Linear Regression (MLR), the result we developed when we cooperated with the local utility company, and the Support Vector Machine (SVM). After thoroughly analyzing and testing the methods based on different parameters, the methods are compared with Mean Square Error (MSE), Root Mean Square Error (RMSE), and Normalized Mean Square Error (NMSE) indexes. The BART displays the best prediction accuracy for every index in our case and its uncertain estimate further provides the confidence interval for the peak load prediction, which also has very high accuracy. Thus, the BART provides the local utility with a better reference peak load forecasting model. Last, influential weather and human factors are summarized.

A machine learning technique merging Bayesian method called Bayesian Additive Regression Trees (BART) provides a nonparametric Bayesian approach that further needs improved forecasting accuracy in the presence of outliers, especially when dealing with potential nonlinear relationships and complex interactions among the response and explanatory variables, which poses a major challenge in forecasting. This study proposes an adaptive trimmed regression method using BART, dubbed BART(Atr) to improve forecasting accuracy by identifying suspected outliers effectively and removing these outliers in the analysis. Through extensive simulations across various scenarios, the effectiveness of BART(Atr) is evaluated against three alternative methods: default BART, robust linear modeling with Huber’s loss function, and data-driven robust regression with Huber’s loss function. The simulation results consistently show BART(Atr) outperforming the other three methods. To demonstrate its practical application, BART(Atr) is applied to the well-known Boston Housing Price dataset, a standard regression analysis example. Furthermore, random attack templates are introduced on the dataset to assess BART(Atr)’s performance under such conditions.

embarcadero is an r package of convenience tools for species distribution modelling (SDM) with Bayesian additive regression trees (BART), a powerful machine learning approach that has been rarely applied to ecological problems. Like other classification and regression tree methods, BART estimates the probability of a binary outcome based on a set of decision trees. Unlike other methods, BART iteratively generates sets of trees based on a set of priors about tree structure and nodes, and builds a posterior distribution of estimated classification probabilities. So far, BARTs have yet to be applied to SDM. embarcadero is a workflow wrapper for BART species distribution models, and includes functionality for easy spartial prediction, an automated variable selection procedure, several types of partial dependence visualization and other tools for ecological application. The embarcadero package is an open source and available on Github. To show how embarcadero can be used by ecologists, I illustrate a BART workflow for a virtual species distribution model. The supplement includes a more advanced vignette showing how BART can be used for mapping disease transmission risk, using the example of Crimean–Congo haemorrhagic fever in Africa.

Bayesian additive regression trees (BART) is a flexible prediction model/machine learning approach that has gained widespread popularity in recent years. As BART becomes more mainstream, there is an increased need for a paper that walks readers through the details of BART, from what it is to why it works. This tutorial is aimed at providing such a resource. In addition to explaining the different components of BART using simple examples, we also discuss a framework, the General BART model that unifies some of the recent BART extensions, including semiparametric models, correlated outcomes, and statistical matching problems in surveys, and models with weaker distributional assumptions. By showing how these models fit into a single framework, we hope to demonstrate a simple way of applying BART to research problems that go beyond the original independent continuous or binary outcomes framework.

Optimizing organ yield (number of organs transplanted per donor) is a modifiable way to increase the number of organs available for transplant. Historically, models to predict donor organ yield have been developed based ordinary least squares regression and ordinal logistic regression; however, alternative modeling methodology may be superior to conventional approaches.1,2 In this preliminary analysis, rather than treating organ yield as a continuous outcome, we modeled the number of organs transplanted per donor as counts. We aimed to compare different linear and nonlinear statistical models for count responses to predict deceased donor organ yield. We used data from the OPTN database from 2000 to 2016 to parameterize our exploratory models. The initial set of predictors for deceased donor organ yield was derived from published studies.1-3 We included adult deceased donors between 18 and 84 years of age that had at least 1 organ procured for transplantation. 75 350 records met inclusion criteria. We used 80% of the data for derivation in a cross-validation analysis and the remainder of the data as a validation set. The cross-validation analysis was replicated 50 times, and the random holdouts consisted of 20% of the derivation cohort. The following models were evaluated: ordinary least squares regression,1 ordinal logistic regression,2 Poisson regression, negative binomial regression, general additive models, classification and regression trees, random forests, bootstrap aggregated classification and regression trees, boosted classification and regression trees, Bayesian additive regression trees (BART), multivariate adaptive regression splines, artificial neural networks, and mean-only models. Among the models, BART resulted in the lowest error on predicting the number of organs transplanted per deceased donor. Two-sample t tests showed that the BART had significantly lower mean absolute error (MAE) when predicting deceased donor organ yield (all P < 0.001). On average, this model presented a MAE of 0.867 throughout the cross-validation analysis, and a MAE of 0.856 when tested in the validation set. The BART showed that deceased donor organ yield had a negative nonlinear relationship with age, body mass index, terminal blood urea nitrogen, terminal laboratory creatinine, aspartate aminotransferase, terminal laboratory total bilirubin; a positive nonlinear relationship with organ recovery time, partial pressure of oxygen levels, and last serum sodium; and more complex nonlinear relationships with alanine aminotransferase and the ratio of partial pressure arterial oxygen and fraction of inspired oxygen. Bayesian additive regression trees would improve prediction from at least 63 organs per 1000 donors (compared with an ordinary least squares regression1) to at most 120 organs per 1000 donors (compared with an ordinal logistic regression2). Through the use of BART, we were able to obtain higher predictive accuracy for organ yield. This model allows for nonlinear relationships among the predictors and the number of organs transplanted per deceased donor, which likely explains the superior performance compared with conventional models. In conclusion, our preliminary analysis shows that the BART methodology is superior in predicting deceased donor organ yield and can potentially serve as an aid to assess organ procurement organization performance, reduce geographic disparities, and in forecasting future organ availability. A forthcoming article will include the finalized analysis.

Statistical comparison of additive regression tree methods on ecological grassland data

BackgroundVarious methods have been developed to investigate the complex and collective effects of environmental mixtures on human health. Tree ensemble methods, such as Bayesian Additive Regression Trees (BART), are known for their stability and accuracy in variable selection and outcome prediction for high-dimensional correlated data in the statistical literature, but their use has not been well studied for environmental mixtures.MethodsWe tailored the original BART model for environmental mixtures analysis to achieve both robust identification of toxic agents and accurate prediction of health outcomes. Our modified BART approach allowed for a smooth response surface and incorporated covariate adjustment for both continuous and binary outcomes. It supported both component-wise variable selection and hierarchical variable selection to accommodate scientifically meaningful groupings of chemicals. To facilitate interpretation, we used a Generalized Additive Model (GAM) approximation to quantify the marginal contributions of individual chemicals. The performance of the modified BART was evaluated through simulations and a case study with the National Health and Nutrition Examination Survey (NHANES) 2001–2002 data to examine the effects of persistent organic pollutants (POPs) on leukocyte telomere length. All results were compared with the Bayesian Kernel Machine Regression (BKMR), a widely used method in mixtures analysis.ResultsOur simulation studies demonstrated that the modified BART produced results comparable to or superior to BKMR in recovering the true exposure-response surface for both continuous and binary outcomes, with consistently above 0.7. Specifically, when chemical groups were considered, modified BART with hierarchical variable selection achieved higher (0.82–0.99 for continuous outcomes and 0.73–0.95 for binary outcomes) than BKMR (0.59–0.67 and 0.47–0.59, respectively), on independent test datasets. Modified BART also reduced the computational time by 70% to 99.8% compared to BKMR. Both methods effectively identified relevant chemical groups under hierarchical variable selection, but modified BART more effectively distinguished important components within groups. In the NHANES case study, three chemicals, including 2,3,4,7,8-pncdf, PCB126 and PCB169, were identified by modified BART as having near-linear positive effects on leukocyte telomere length based on GAM approximation plots.ConclusionsModified BART is a robust and scalable response surface model alternative to BKMR for analyzing environmental mixtures data. It is particularly advantageous for large datasets, binary outcomes, and grouped chemicals. GAM approximation provides practical insights into interpreting individual chemical effect estimated from complex response surface models.

Bayesian additive regression trees (BART) is a fully Bayesian approach to modeling with ensembles of trees. BART can uncover complex regression functions with high-dimensional regressors in a fairly automatic way and provide Bayesian quantification of the uncertainty through the posterior. However, BART assumes independent and identical distributed (i.i.d) normal errors. This strong parametric assumption can lead to misleading inference and uncertainty quantification. In this chapter we use the classic Dirichlet process mixture (DPM) mechanism to nonparametrically model the error distribution. A key strength of BART is that default prior settings work reasonably well in a variety of problems. The challenge in extending BART is to choose the parameters of the DPM so that the strengths of the standard BART approach is not lost when the errors are close to normal, but the DPM has the ability to adapt to non-normal errors.

Tree-based regression and classification has become a standard tool in modern data science. Bayesian Additive Regression Trees (BART) has in particular gained wide popularity due its flexibility in dealing with interactions and non-linear effects. BART is a Bayesian tree-based machine learning method that can be applied to both regression and classification problems and yields competitive or superior results when compared to other predictive models. As a Bayesian model, BART allows the practitioner to explore the uncertainty around predictions through the posterior distribution. In this paper, we present new Visualisation techniques for exploring BART models. We construct conventional plots to analyse a model’s performance and stability as well as create new tree-based plots to analyse variable importance, interaction, and tree structure. We employ Value Suppressing Uncertainty Palettes (VSUP) to construct heatmaps that display variable importance and interactions jointly using colour scale to represent posterior uncertainty. Our new Visualisations are designed to work with the most popular BART R packages available, namely BART, dbarts, and bartMachine. Our approach is implemented in the R package bartMan (BART Model ANalysis).

Estimation of individualized treatment effects (ITE), also known as conditional average treatment effects (CATE), is an active area of methodology development. However, much less attention has been paid to the quantification of uncertainty of ITE/CATE estimates in right-censored survival data. Here we undertake an extensive simulation study to examine the coverage of interval estimates from two popular estimation algorithms, Bayesian additive regression trees (BART) and causal survival forest (CSF). We conducted simulation designs from 3 different settings: first, in a setting where BART was developed for an accelerated failure time model; second, where CSF was developed; and finally, a “neutral” simulation taken from a setting where neither BART nor CSF was developed. BART outperformed CSF in all three simulation settings. Both the BART and CSF algorithms involve multiple hyperparameters, and BART credible intervals had better coverage than the CSF confidence intervals under the default values, as well as under optimized values, of these hyperparameters.