Bayesian Additive Regression Trees Research Articles

Mortality Prediction After Cardiac Surgery in Children: An STS Congenital Heart Surgery Database Analysis

BackgroundThe Society of Thoracic Surgeons (STS) Congenital Heart Surgery Database (CHSD) provides risk-adjusted operative mortality rates to approximately 120 North American congenital heart centers. Optimal case-mix adjustment methods for operative mortality risk prediction in this population remain unclear. MethodsA panel created diagnosis-procedure combinations of encounters in the CHSD. Models for operative mortality using the new diagnosis-procedure categories, procedure-specific risk factors, and syndromes or abnormalities included in the CHSD were estimated using Bayesian additive regression trees and least absolute shrinkage and selector operator (lasso) models. Performance of the new models was compared with the current STS CHSD risk model. ResultsOf 98 825 operative encounters (69 063 training; 29 762 testing), 2818 (2.85%) STS-defined operative mortalities were observed. Differences in sensitivity, specificity, and true and false positive predicted values were negligible across models. Calibration for mortality predictions at the higher end of risk from the lasso and Bayesian additive regression trees models was better than predictions from the STS CHSD model, likely because of the new models’ inclusion of diagnosis-palliative procedure variables affecting <1% of patients overall but accounting for 27% of mortalities. Model discrimination varied across models for high-risk procedures, hospital volume, and hospitals. ConclusionsOverall performance of the new models did not differ meaningfully from the STS CHSD risk model. Adding procedure-specific risk factors and allowing diagnosis to modify predicted risk for palliative operations may augment model performance for very high-risk surgical procedures. Given the importance of risk adjustment in estimating hospital quality, a comparative assessment of surgical program quality evaluations using the different models is warranted.

Patent Analysis Using Bayesian Data Analysis and Network Modeling

Patent analysis is to analyze patent data to understand target technology. Patent data contains various detailed information about the developed technology. Therefore, many studies concerning patent analysis have been carried out in the technology analysis fields. Most traditional methods for technology analysis were based on qualitative approaches such as Delphi survey. However, the patent analysis methods based on statistics and machine learning have been introduced recently. In this paper, we proposed a statistical method for quantitative patent analysis. Moreover, we selected drone technology as the target technology for patent analysis. To understand drone technology, we analyzed the patents on drone technology. We searched the patent documents related to drone technology and transformed them to structured data using text mining techniques. First, we visualized the patent keywords to identify the technological structure of a drone. Next, using Bayesian additive regression trees, we analyzed the structured patent data to construct technology scenarios for drones. To illustrate the performance and validity of our proposed research, we presented the experimental results of patent analysis using patent documents related to drone technology.

Using BART to Perform Pareto Optimization and Quantify its Uncertainties

Techniques to reduce the energy burden of an industrial ecosystem often require solving a multiobjective optimization problem. However, collecting experimental data can often be either expensive or time-consuming. In such cases, statistical methods can be helpful. This article proposes Pareto Front (PF) and Pareto Set (PS) estimation methods using Bayesian Additive Regression Trees (BART), which is a nonparametric model whose assumptions are typically less restrictive than popular alternatives, such as Gaussian Processes (GPs). These less restrictive assumptions allow BART to handle scenarios (e.g., high-dimensional input spaces, nonsmooth responses, large datasets) that GPs find difficult. The performance of our BART-based method is compared to a GP-based method using analytic test functions, demonstrating convincing advantages. Finally, our BART-based methodology is applied to a motivating engineering problem. Supplementary materials, which include a theorem proof, algorithms, and R code, for this article are available online.

Inference from Nonrandom Samples Using Bayesian Machine Learning.

We consider inference from nonrandom samples in data-rich settings where high-dimensional auxiliary information is available both in the sample and the target population, with survey inference being a special case. We propose a regularized prediction approach that predicts the outcomes in the population using a large number of auxiliary variables such that the ignorability assumption is reasonable and the Bayesian framework is straightforward for quantification of uncertainty. Besides the auxiliary variables, we also extend the approach by estimating the propensity score for a unit to be included in the sample and also including it as a predictor in the machine learning models. We find in simulation studies that the regularized predictions using soft Bayesian additive regression trees yield valid inference for the population means and coverage rates close to the nominal levels. We demonstrate the application of the proposed methods using two different real data applications, one in a survey and one in an epidemiologic study.

Hourly streamflow forecasting using a Bayesian additive regression tree model hybridized with a genetic algorithm

Urban flooding is a global metropolitan problem; therefore, establishing reliable streamflow forecasting models is critical for flood control and planning in urban areas. Furthermore, assessing the importance and uncertainty of model predictors is useful for managers; however, these predictors are still underevaluated. To address these concerns, we developed a novel hybrid model, GA-BART, that integrates a genetic algorithm (GA) with the Bayesian additive regression tree (BART) model for hourly streamflow forecasting. A case study was conducted in the Jungrang urban basin, which is located on the Han River in South Korea. The model was built and evaluated based on data collected during 39 heavy rain events from 2003 to 2020. To compare the model’s forecast capability, a support vector regression model hybridized with a genetic algorithm (GA-SVR) and a multiple linear regression (MLR) model was constructed. An analysis of multiple datasets including different input predictors was performed to define the optimal set for streamflow forecasting. The results illustrated that the GA-BART model outperformed the GA-SVR and MLR models in multistep-ahead streamflow forecasts, with improved measures of the root mean square error (RMSE), mean absolute error (MAE), relative error, Nash-Sutcliffe efficiency (NSE), time lag and correlation coefficient (CC). In addition, the GA-BART model could reasonably determinethe relative importance of the input variables. This study demonstrated that, despite some disadvantages in the five- and six-hour step-ahead forecasts, the hybrid GA-BART model can be a good option among the available models for hourly streamflow forecasting.

Kolon Kanserini Öngörmede Bayesian Eklemeli Regresyon Ağaçları: Metodolojik Çalışma (Geçerlik Çalışması

Objective: The occurrence of colon cancer starts in the inner wall of the large intestine. The survival of colon cancer patients strongly relies on early detection. Diagnosing colon cancer using clinical approaches often takes longer, especially in most developing countries with limited facilities. The recent use of microarray technology has presented a new approach for the oncologist to diagnose cancer cells using non-clinical machine learning methods. In this paper, the aim is to predict the status of colon cancer tissues using the Bayesian Additive Regression Trees (BART) and 2 other machine learning methods. Material and Methods: The development and comparative analysis of BART alongside 2 other competing methods (Random Forest: RF and Gradient Boosting Machine: GBM) were implemented. The dataset used for the analysis is the microarray colon cancer data which consists of 2,000 gene expression measurements for 62 tissue samples. Results: The methods are compared based on overall metrics (accuracy, balance accuracy, detection rate, F-measure and AUC) and class-specific metrics (sensitivity, specificity, positive predictive value and negative predictive value). The overall metrics results showed that the best method is RF. The class-specific metrics results showed that BART is better than RF. Conclusion: On average, BART is more sensitive in detecting the presence of colon cancer cells, while RF is more accurate and specific in detecting the presence or absence of colon cancer cells.

Investigate the causal effect of diversification strategy on risk-adjusted performance using Bayesian additive regression trees

Investigate the causal effect of diversification strategy on risk-adjusted performance using Bayesian additive regression trees

Investigate the causal effect of diversification strategy on risk-adjusted performance using Bayesian additive regression trees

Investigate the causal effect of diversification strategy on risk-adjusted performance using Bayesian additive regression trees

Impulse response estimation via flexible local projections

This paper introduces a flexible local projection that generalizes the model by Jord\'a (2005) to a non-parametric setting using Bayesian Additive Regression Trees. Monte Carlo experiments show that our BART-LP model is able to capture non-linearities in the impulse responses. Our first application shows that the fiscal multiplier is stronger in recession than in expansion only in response to contractionary fiscal shocks, but not in response to expansionary fiscal shocks. We then show that financial shocks generate effects on the economy that increase more than proportionately in the size of the shock when the shock is negative, but not when the shock is positive.

Understanding electricity billing preferences in rural and urban India: Evidence from a conjoint experiment

To address issues of non-payment, high costs, and theft, paying a fixed fee for electricity is common among many developing countries. We use a conjoint experiment to study electricity billing preferences among urban and rural communities in Uttar Pradesh, India. We find that 59.5% of respondents (95% CI: 58.2%–60.9%) prefer consumption-based tariffs as opposed to fixed fee ones, favoring lower base charges among a number of factors. We additionally use Bayesian Additive Regression Trees to test for heterogeneous treatment effects. Respondents with more appliances, using more hours of electricity, and who live in rural areas with meters prefer consumption-based plans with lower base rates. Our results suggest that policy reforms should move beyond fixed rate schemes especially if respondents would accept higher unit tariffs with improved service.

Bayesian optimization with adaptive surrogate models for automated experimental design

Bayesian optimization (BO) is an indispensable tool to optimize objective functions that either do not have known functional forms or are expensive to evaluate. Currently, optimal experimental design is always conducted within the workflow of BO leading to more efficient exploration of the design space compared to traditional strategies. This can have a significant impact on modern scientific discovery, in particular autonomous materials discovery, which can be viewed as an optimization problem aimed at looking for the maximum (or minimum) point for the desired materials properties. The performance of BO-based experimental design depends not only on the adopted acquisition function but also on the surrogate models that help to approximate underlying objective functions. In this paper, we propose a fully autonomous experimental design framework that uses more adaptive and flexible Bayesian surrogate models in a BO procedure, namely Bayesian multivariate adaptive regression splines and Bayesian additive regression trees. They can overcome the weaknesses of widely used Gaussian process-based methods when faced with relatively high-dimensional design space or non-smooth patterns of objective functions. Both simulation studies and real-world materials science case studies demonstrate their enhanced search efficiency and robustness.

Comparison of machine learning clustering algorithms for detecting heterogeneity of treatment effect in acute respiratory distress syndrome: A secondary analysis of three randomised controlled trials.

BackgroundHeterogeneity in Acute Respiratory Distress Syndrome (ARDS), as a consequence of its non-specific definition, has led to a multitude of negative randomised controlled trials (RCTs). Investigators have sought to identify heterogeneity of treatment effect (HTE) in RCTs using clustering algorithms. We evaluated the proficiency of several commonly-used machine-learning algorithms to identify clusters where HTE may be detected.MethodsFive unsupervised: Latent class analysis (LCA), K-means, partition around medoids, hierarchical, and spectral clustering; and four supervised algorithms: model-based recursive partitioning, Causal Forest (CF), and X-learner with Random Forest (XL-RF) and Bayesian Additive Regression Trees were individually applied to three prior ARDS RCTs. Clinical data and research protein biomarkers were used as partitioning variables, with the latter excluded for secondary analyses. For a clustering schema, HTE was evaluated based on the interaction term of treatment group and cluster with day-90 mortality as the dependent variable.FindingsNo single algorithm identified clusters with significant HTE in all three trials. LCA, XL-RF, and CF identified HTE most frequently (2/3 RCTs). Important partitioning variables in the unsupervised approaches were consistent across algorithms and RCTs. In supervised models, important partitioning variables varied between algorithms and across RCTs. In algorithms where clusters demonstrated HTE in the same trial, patients frequently interchanged clusters from treatment-benefit to treatment-harm clusters across algorithms. LCA aside, results from all other algorithms were subject to significant alteration in cluster composition and HTE with random seed change. Removing research biomarkers as partitioning variables greatly reduced the chances of detecting HTE across all algorithms.InterpretationMachine-learning algorithms were inconsistent in their abilities to identify clusters with significant HTE. Protein biomarkers were essential in identifying clusters with HTE. Investigations using machine-learning approaches to identify clusters to seek HTE require cautious interpretation.FundingNIGMS R35 GM142992 (PS), NHLBI R35 HL140026 (CSC); NIGMS R01 GM123193, Department of Defense W81XWH-21-1-0009, NIA R21 AG068720, NIDA R01 DA051464 (MMC)

Ensemble Modeling on Near-Infrared Spectra as Rapid Tool for Assessment of Soil Health Indicators for Sustainable Food Production Systems

A novel total ensemble (TE) algorithm was developed and compared with random forest optimization (RFO), gradient boosted machines (GBM), partial least squares (PLS), Cubist and Bayesian additive regression tree (BART) algorithms to predict numerous soil health indicators in soils with diverse climate-smart land uses at different soil depths. The study investigated how land-use practices affect several soil health indicators. Good predictions using the ensemble method were obtained for total carbon (R2 = 0.87; RMSE = 0.39; RPIQ = 1.36 and RPD = 1.51), total nitrogen (R2 = 0.82; RMSE = 0.03; RPIQ = 2.00 and RPD = 1.60), and exchangeable bases, m3. Cu, m3. Fe, m3. B, m3. Mn, exchangeable Na, Ca (R2 &gt; 0.70). The performances of algorithms were in order of TE &gt; Cubist &gt; BART &gt; PLS &gt; GBM &gt; RFO. Soil properties differed significantly among land uses and between soil depths. In Kenya, however, soil pH was not significant, except at depths of 45–100 cm, while the Fe levels in Tanzanian grassland were significantly high at all depths. Ugandan agroforestry had a substantially high concentration of ExCa at 0–15 cm. The total ensemble method showed better predictions as compared to other algorithms. Climate-smart land-use practices to preserve soil quality can be adopted for sustainable food production systems.

Prediction of groundwater nitrate concentration in a semiarid region using hybrid Bayesian artificial intelligence approaches.

Nitrate is a major pollutant in groundwater whose main source is municipal wastewater and agricultural activities. In the present study, Bayesian approaches such as Bayesian generalized linear model (BGLM), Bayesian regularized neural network (BRNN), Bayesian additive regression tree (BART), and Bayesian ridge regression (BRR) were used to model groundwater nitrate contamination in a semiarid region Marvdasht watershed, Fars province, Iran. Eleven groundwater (GW) nitrate conditioning factors have been taken as input parameters for predictive modeling. The results showed that the Bayesian models used in this study were all competent to model groundwater nitrate and the BART model with R2 = 0.83 was more efficient than the other models. The result of variable importance showed that potassium (K) has the highest importance in the models followed by rainfall, altitude, groundwater depth, and distance from the residential area. The results of the study can support the decision-making process to control and reduce the sources of nitrate pollution.

Multiple imputation procedures for estimating causal effects with multiple treatments with application to the comparison of healthcare providers.

Choosing between multiple healthcare providers requires us to simultaneously compare the expected outcomes under each provider. This comparison is complex because the composition of patients treated by each provider may differ. Similar issues arise when simultaneously comparing the adverse effects of interventions using non-randomized data. To simultaneously estimate the effects of multiple providers/interventions we propose procedures that explicitly impute the set of potential outcomes for each subject. The procedures are based on different specifications of the generalized additive models (GAM) and the Bayesian additive regression trees (BART). We compare the performance of the proposed procedures to previously proposed matching and weighting procedures using an extensive simulation study for continuous outcomes. Our simulations show that when the distributions of the covariates across treatment groups have adequate overlap, the multiple imputation procedures based on separate BART or GAM models in each treatment group are generally superior to weighting based methods and have similar and sometimes better performance than matching on the logit of the generalized propensity score. Another advantage of these multiple imputation procedures is the ability to provide point and interval estimates to a wide range of causal effect estimands. We apply the proposed procedures to comparing multiple nursing homes in Massachusetts for readmission outcomes. The proposed approach can be applied to other causal effects applications with multiple treatments.

Machine Learning for Prediction of Stable Warfarin Dose in US Latinos and Latin Americans.

Populations used to create warfarin dose prediction algorithms largely lacked participants reporting Hispanic or Latino ethnicity. While previous research suggests nonlinear modeling improves warfarin dose prediction, this research has mainly focused on populations with primarily European ancestry. We compare the accuracy of stable warfarin dose prediction using linear and nonlinear machine learning models in a large cohort enriched for US Latinos and Latin Americans (ULLA). Each model was tested using the same variables as published by the International Warfarin Pharmacogenetics Consortium (IWPC) and using an expanded set of variables including ethnicity and warfarin indication. We utilized a multiple linear regression model and three nonlinear regression models: Bayesian Additive Regression Trees, Multivariate Adaptive Regression Splines, and Support Vector Regression. We compared each model’s ability to predict stable warfarin dose within 20% of actual stable dose, confirming trained models in a 30% testing dataset with 100 rounds of resampling. In all patients (n = 7,030), inclusion of additional predictor variables led to a small but significant improvement in prediction of dose relative to the IWPC algorithm (47.8 versus 46.7% in IWPC, p = 1.43 × 10−15). Nonlinear models using IWPC variables did not significantly improve prediction of dose over the linear IWPC algorithm. In ULLA patients alone (n = 1,734), IWPC performed similarly to all other linear and nonlinear pharmacogenetic algorithms. Our results reinforce the validity of IWPC in a large, ethnically diverse population and suggest that additional variables that capture warfarin dose variability may improve warfarin dose prediction algorithms.

A comparative study of design-based and analysis-based approaches to causal inference with observational data

Causal inference with observational data is a central goal in many fields. Propensity score methods are design-based approaches that try to ensure covariate balance without using information from the outcome variables. Analysis-based approaches, such as the Bayesian Additive Regression Tree and the Causal Forest, bypass the issue of covariate balance, and directly model the outcomes. We use a Monte Carlo simulation to study the performance of these two types of approaches. Some of the simulation scenarios involve large number of covariates relative to the number of observations. We find that the analysis-based approaches can yield very poor performance, without any warning about not enough overlap between the covariate distributions for the treated and control groups. In contrast, the propensity score methods provide warning about not enough overlap, but such warning could be overly-cautious when there is enough overlap.

Variable selection with missing data in both covariates and outcomes: Imputation and machine learning.

Variable selection in the presence of both missing covariates and outcomes is an important statistical research topic. Parametric regression are susceptible to misspecification, and as a result are sub-optimal for variable selection. Flexible machine learning methods mitigate the reliance on the parametric assumptions, but do not provide as naturally defined variable importance measure as the covariate effect native to parametric models. We investigate a general variable selection approach when both the covariates and outcomes can be missing at random and have general missing data patterns. This approach exploits the flexibility of machine learning models and bootstrap imputation, which is amenable to nonparametric methods in which the covariate effects are not directly available. We conduct expansive simulations investigating the practical operating characteristics of the proposed variable selection approach, when combined with four tree-based machine learning methods, extreme gradient boosting, random forests, Bayesian additive regression trees, and conditional random forests, and two commonly used parametric methods, lasso and backward stepwise selection. Numeric results suggest that, extreme gradient boosting and Bayesian additive regression trees have the overall best variable selection performance with respect to the score and Type I error, while the lasso and backward stepwise selection have subpar performance across various settings. There is no significant difference in the variable selection performance due to imputation methods. We further demonstrate the methods via a case study of risk factors for 3-year incidence of metabolic syndrome with data from the Study of Women's Health Across the Nation.

Improved Prediction of Older Adult Discharge After Trauma Using a Novel Machine Learning Paradigm

The ability to reliably predict outcomes after trauma in older adults (age ≥ 65 y) is critical for clinical decision making. Using novel machine-learning techniques, we sought to design a nonlinear, competing risks paradigm for prediction of older adult discharge disposition following injury. The National Trauma Databank (NTDB) was used to identify patients 65+ y between 2007 and 2014. Training was performed on an enriched cohort of diverse patients. Factors included age, comorbidities, length of stay, and physiologic parameters to predict in-hospital mortality and discharge disposition (home versus skilled nursing/long-term care facility). Length of stay and discharge status were analyzed via competing risks survival analysis with Bayesian additive regression trees and a multinomial mixed model. The resulting sample size was 47,037 patients. Admission GCS and age were important in predicting mortality and discharge disposition. As GCS decreased, patients were more likely to die (risk ratio increased by average of 1.4 per 2-point drop in GCS, P < 0.001). As GCS decreased, patients were also more likely to be discharged to a skilled nursing or long-term care facility (risk ratio decreased by 0.08 per 2-point decrease in GCS, P< 0.001). The area under curve for prediction of discharge home was improved in the competing risks model 0.73 versus 0.43 in the traditional multinomial mixed model. Predicting older adult discharge disposition after trauma is improved using machine learning over traditional regression analysis. We confirmed that a nonlinear, competing risks paradigm enhances prediction on any given hospital day post injury.

Novel Bayesian Additive Regression Tree Methodology for Flood Susceptibility Modeling

Identifying areas prone to flooding is a key step in flood risk management. The purpose of this study is to develop and present a novel flood susceptibility model based on Bayesian Additive Regression Tree (BART) methodology. The predictive performance of the new model is assessed via comparison with the Naïve Bayes (NB) and Random Forest (RF) based methods that were previously published in the literature. All models were tested on a real case study based in the Kan watershed in Iran. The following fifteen climatic and geo-environmental variables were used as inputs into all flood susceptibility models: altitude, aspect, slope, plan curvature, profile curvature, drainage density, distance from river distance from road, stream power index (SPI), topographic wetness index (TPI), topographic position index (TPI), curve number (CN), land use, lithology and rainfall. Based on the existing flood field survey and other information available for the analyzed area, a total of 118 flood locations were identified as potentially prone to flooding. The data available were divided into two groups with 70% used for training and 30% for validation of all models. The receiver operating characteristic (ROC) curve parameters were used to evaluate the predictive accuracy of the new and existing models. Based on the area under curve (AUC) the new BART (86%) model outperformed the NB (80%) and RF (85%) models. Regarding the importance of input variables, the results obtained showed that the location’s altitude and distance from the river are the most important variables for assessing flooding susceptibility.