Tree-based Machine Learning Models Research Articles

Division Does Not Imply Predictability: Demographics Continue to Reveal Little About Voting and Partisanship

What are the political consequences of ongoing social sorting? We evaluate the degree of social sorting and mass polarization using the predictability of partisanship and voting decisions as quantities of interest. Contrary to expectations, demographic sorting has not produced a very predictable electorate. Models trained on nothing more than demographic labels from public opinion surveys (1952–2020) predict only 63.9% of two-party vote choices and 63.4% of partisan IDs correctly out-of-sample—whether they be based on logistic regressions or tree-based machine learning models. Moreover, demographics’ predictive power over vote choice or partisan affiliation shows a surprising stability over time. We argue that while select demographics’ marginal effects may appear to be evidence of social sorting, the joint predictability of political behavior using only demographic characteristics has been, and still is, modest at best.

Tree-based machine learning models for prediction of bed elevation around bridge piers

Scouring around bridge piers is a highly nonlinear process making its prediction by deterministic and stochastic models challenging. This study explores the application of inferential models for predictions of bed elevations around bridge piers. The objective is to get a generalized machine learning model with an interpretable structure. The historical data comprise a detailed record of streamflow and bed elevations that were captured by sensors installed at the 5th Street Bridge piers over Ocmulgee River at Macon, GA. We investigate the accuracy and efficiency of various tree-based machine learning algorithms, including a single tree as well as homogeneous ensemble models for simultaneous predictions of bed elevation at multiple sensors installed at piers. The ensemble models were based on bagging and boosting techniques. Special attention is given to balancing between overfitting and underfitting without compromise on the model's robustness. Observation of the performance metrics showed that tree-based models have excellent predictive capacity. It was observed that boosting models, including a gradient based regression model, and adaptive boosting outperformed the bagging model. Among all the models investigated in this study, the adaptive boosting method was observed to be most generalizable. The performance of developed models shows the potential of tree-based ensemble models in providing rapid and robust predictions for complex nonlinear fluid flows.

Applicability of decision tree-based machine learning models in the prediction of core-calibrated shale facies from wireline logs in the late Devonian Duvernay Formation, Alberta, Canada

Well logs provide insight into stratigraphically compartmentalized rock properties and are a cost-effective alternative to core. The identification of reservoir (and nonreservoir) facies in core, and their calibration to well-log response has traditionally relied on expert domain knowledge and is inherently inconsistent. Such analyses are time-consuming, tedious, error prone, and often biased due to a lack of objectivity. Automated lithologic interpretations from wireline logs appear to be a promising solution for identifying and understanding depositional complexity within a reservoir. Using the Duvernay Formation in the Western Canada Sedimentary Basin as a case study, the authors evaluate the applicability of decision tree-based machine learning (ML) methods in the prediction of core-calibrated facies and/or facies association distributions within wireline logs. The authors use three independent decision tree-based ML models to predict (1) facies (FACM), (2) facies associations (FAM), and (3) reservoir rock (RESM) from wireline logs. Model accuracies are 60.3%, 88.1%, and 88.1% for FACM, FAM, and RESM, respectively, but individual class F1 scores range from 0 to 0.92. The authors attribute discrepancies in individual class performance to interval thickness, sample proportion of training data, and distinguishability of the output class. Classes thicker than 3 m and encompassing at least 16% of the training data set have F1 scores greater than 0.60. The authors attribute exceptions to these general cutoffs to the ability to recognize diagnostic sedimentologic features observed in core. Results from this study help in understanding stratigraphic complexity in the absence of core aiding in subsurface characterization of reservoirs.

C2PAx: Complexity-Aware Constant Parameter Approximation for Energy-Efficient Tree-Based Machine Learning Accelerators

Tree-based machine learning models, like random forests and decision trees, are low-complexity solutions that provide an efficient prediction for a wide range of applications. These models are particularly interesting for energy-constrained platforms since they can be implemented with simple logical operations. Tree-based accelerators are also intrinsically resilient to errors, and this can be leveraged to boost energy efficiency with approximate computing techniques. The key operations in these models are comparisons to constants, making comparators excellent candidates for approximation. This paper presents a technique to approximate comparisons to constants called C2PAx, which is capable of reducing the area and energy of tree-based accelerators. The method consists in finding alternative constants that reduce circuit area while keeping an efficient prediction performance. It is also shown that the selection of the constant parameters directly influences both hardware complexity and model performance, demanding cross-layer optimization. For that, we extend an existing framework that generates VLSI tree-based accelerators, inserting our approximation proposal that allows selecting the constant parameters that maximize energy efficiency at the cost of minor accuracy drops. Simulation results demonstrate that C2PAx outperforms the Don’t Care logic approximation technique when accuracy and energy are jointly considered. C2PAx trades accuracy for significant reductions in the VLSI area, power, delay, and energy consumption compared to precise models.

Predicting academic performance using tree-based machine learning models: A case study of bachelor students in an engineering department in China

Predicting academic performance using tree-based machine learning models: A case study of bachelor students in an engineering department in China

Discussion on the tree-based machine learning model in the study of landslide susceptibility

Discussion on the tree-based machine learning model in the study of landslide susceptibility

The impact of tree-based machine learning models, length of training data, and quarantine search query on tourist arrival prediction’s accuracy under COVID-19 in Indonesia

ABSTRACT This study presents the extreme gradient boosting (XGBoost) and random forest (RF) models to predict tourism demand by incorporating international COVID-19 cases, international tourist arrivals, and the destination's quarantine policy predictors. Unlike other ‘black box’ machine learning models, those two tree-based models offer better interpretability with explicit feature importance and tree structure representations. This paper evaluates the accuracy of these models in predicting international tourist arrivals in Indonesia during the COVID-19 pandemic using long-range (January 2008–June 2021) and short-range (January 2018–June 2021) training datasets. The performance of these two models is compared with benchmark models, such as the artificial neural network, autoregressive integrated moving average, and seasonal ARIMA models. In general, the tree-based machine learning models outperformed all benchmark models. International COVID-19 cases and tourist arrivals predictors have dominating feature importance scores in XGBoost models. Meanwhile, Google trends keywords on quarantine policies show significant importance in RF models but not in the XGBoost models. Moreover, RF models are better than the XGBoost models in terms of accuracy and overcoming overfitting cases.

Predicting Bulk Average Velocity with Rigid Vegetation in Open Channels Using Tree-Based Machine Learning: A Novel Approach Using Explainable Artificial Intelligence.

Predicting the bulk-average velocity (UB) in open channels with rigid vegetation is complicated due to the non-linear nature of the parameters. Despite their higher accuracy, existing regression models fail to highlight the feature importance or causality of the respective predictions. Therefore, we propose a method to predict UB and the friction factor in the surface layer (fS) using tree-based machine learning (ML) models (decision tree, extra tree, and XGBoost). Further, Shapley Additive exPlanation (SHAP) was used to interpret the ML predictions. The comparison emphasized that the XGBoost model is superior in predicting UB (R = 0.984) and fS (R = 0.92) relative to the existing regression models. SHAP revealed the underlying reasoning behind predictions, the dependence of predictions, and feature importance. Interestingly, SHAP adheres to what is generally observed in complex flow behavior, thus, improving trust in predictions.

Mass appraisal as affordable public policy: Open data and machine learning for mapping urban land values

Updated cadastral land values are a matter of critical importance for local governments: higher revenue of property taxes, more equitable treatment to taxpayers, a fundamental input in the design of public policies related to access to land and housing for the most vulnerable and a key feature in land value capture strategies to finance public infrastructure, to name just a few public policies that require correct valuations of land. However, in Latin America, outdated cadastral values are common to most cities. The reasons for this can be found in the complexity of the mass appraisal process, lack of institutional and fiscal capacity to undertake it and bureaucratic resistance to its implementation.The objective of this paper is to present a mass appraisal methodology that uses only free and open data to achieve robust urban land valuations. Information from the OpenStreetMap Project is used to generate several land variables. In addition, the Global Human Settlement Layer of the European Commission is used to determine the level of consolidation of urban sprawl. Land value data were obtained from the Mapa de Valores de América Latina, a collaborative initiative that systemizes more than 68,000 data from more than 900 cities.This information is used to train three tree-based machine learning models: Random Forest, Quantile Random Forest and Gradient Boosting Model. The results support the viability of the proposed strategy, simplifying the mass appraisal process in terms of costs, time and complexity of the information used.

Assessment of flood susceptibility prediction based on optimized tree-based machine learning models

Abstract Due to the physical processes of floods, the use of data-driven machine learning (ML) models is a cost-efficient approach to flood modeling. The innovation of the current study revolves around the development of tree-based ML models, including Rotation Forest (ROF), Alternating Decision Tree (ADTree), and Random Forest (RF) via binary particle swarm optimization (BPSO), to estimate flood susceptibility in the Maneh and Samalqan watershed, Iran. Therefore, to implement the models, 370 flood-prone locations in the case study were identified (2016–2019). In addition, 20 hydrogeological, topographical, geological, and environmental criteria affecting flood occurrence in the study area were extracted to predict flood susceptibility. The area under the curve (AUC) and a variety of other statistical indicators were used to evaluate the performances of the models. The results showed that the RF-BPSO (AUC=0.935) has the highest accuracy compared to ROF-BPSO (AUC=0.904), and ADTree-BPSO (AUC=0.923). In addition, the findings illustrated that the chance of flooding in the center of the area in question is greater than in other points due to lower elevation, lower slope, and proximity to rivers. Therefore, the ensemble framework proposed here can also be used to predict flood susceptibility maps in other regions with similar geo-environmental characteristics for flood management and prevention.

Forecasting an electricity demand threshold to proactively trigger cost saving demand response actions

This paper presents a novel methodology that empowers virtually any electricity consumer paying for peak demand charges to proactively execute demand response actions even without receiving signals or information coming from the utility, and only when necessary to effectively reduce demand charges and user inconvenience. The proposed methodology employs different arithmetic models and tree-based machine learning models to determine an efficient electricity demand threshold value before the start of a billing period. This methodology is completely model agnostic so additional models can be integrated without changing the proposed process. The threshold value produced can be used to proactively trigger peak demand shaving and other demand response actions in order to reduce demand charges. The results obtained using real data showed that regression random decision forest based models outperformed different arithmetic models and other tree-based machine learning models at determining this threshold value for an industrial, an educational with solar photovoltaic electricity generation, and a residential consumer. The results also showed that the consumers evaluated could potentially achieve between 63% to 75% of total potential demand charge reductions during a year. These results translate to US$ 159.00, US$ 23,290.00, and US$ 107,389.00 in savings for the residential, industrial, and educational consumer respectively.

TREE-BASED MACHINE LEARNING METHODS FOR MODELING AND FORECASTING MORTALITY

AbstractMachine learning has recently entered the mortality literature in order to improve the forecasts of stochastic mortality models. This paper proposes to use two pure, tree-based machine learning models: random forests and gradient boosting, based on the differenced log-mortality rates to produce more accurate mortality forecasts. These forecasts are compared with forecasts from traditional, stochastic mortality models and with forecasts from random forests and gradient boosting variants of the stochastic models. The comparisons are based on the Model Confidence Set procedure. The results show that the pure, tree-based models significantly outperform all other models in the majority of cases considered. To address the lack of interpretability issue associated with machine learning models, we demonstrate how to extract information about the relationships uncovered by the tree-based models. For this purpose, we consider variable importance, partial dependence plots, and variable split conditions. Results from the in-sample fit suggest that tree-based models can be very useful tools for detecting patterns within and between variables that are not commonly identifiable with traditional methods.

Improvement of IMRT QA prediction using imaging-based neural architecture search.

Machine learning (ML) has been used to predict the gamma passing rate (GPR) of intensity-modulated radiation therapy (IMRT) QA results. In this work, we applied a novel neural architecture search to automatically tune and search for the best deep neural networks instead of using hand-designed deep learning architectures. One hundred and eighty-two IMRT plans were created and delivered with portal dosimetry. A total of 1497 fields for multiple treatment sites were delivered and measured by portal imagers. Gamma criteria of 2%/2mm with a 5% threshold were used. Fluence maps calculated for each plan were used as inputs to a convolution neural network (CNN). Auto-Keras was implemented to search for the best CNN architecture for fluence image regression. The network morphism was adopted in the searching process, in which the base models were ResNet and DenseNet. The performance of this CNN approach was compared with tree-based ML models previously developed for this application, using the same dataset. The deep-learning-based approach had 98.3% of predictions within 3% of the measured 2%/2-mm GPRs with a maximum error of 3.1% and a mean absolute error of less than 1%. Our results show that this novel architecture search approach achieves comparable performance to the machine-learning-based approaches with handcrafted features. We implemented a novel CNN model using imaging-based neural architecture for IMRT QA prediction. The imaging-based deep-learning method does not require a manual extraction of relevant features and is able to automatically select the best network architecture.

Artificial intelligence modeling of ultrasonic fatigue test to predict the temperature increase

The temperature behavior in very high cycle fatigue (VHCF) testing as well as the influence of the intermittent loading is not completely understood. In many cases the high frequency causes the specimens to heat up and may interfere in the material's fatigue performance. In order to address this issue, this study proposed an experimental test with different stress levels and intermittent driving (pulse-pause) with the aid of non-destructive testing (NDT) using a thermography camera. Specimens were coated with black spray to improve the emissivity to 0.93 and conducted to fully reversed condition (R = −1) up to 107 cycles. A large amount of raw data of pulse, pause, stress amplitude, number of cycles and temperature were recorded. These raw data were used to develop tree-based machine learning models called extreme gradient boosting (XGBoost), capable of predicting the temperature throughout the VHCF tests. The result presented a high performance model with determination coefficients (R2) above 0.98, proving the model to be an important ally for ultrasonic fatigue tests. Additionally, Shapley additive explanation (SHAP) method was adopted to assist in the interpretation of the model results.

Decision tree-based machine learning models for above-ground biomass estimation using multi-source remote sensing data and object-based image analysis

Forest above-ground biomass (AGB) estimation provides valuable information about the carbon cycle. Thus, the overall goal of this paper is to present an approach to enhance the accuracy of the AGB estimation. The main objectives are to: 1) investigate the performance of remote sensing data sources, including airborne light detection and ranging (LiDAR), optical, SAR, and their combination to improve the AGB predictions, 2) examine the capability of tree-based machine learning models, and 3) compare the performance of pixel-based and object-based image analysis (OBIA). To investigate the performance of machine learning models, multiple tree-based algorithms were fitted to predictors derived from airborne LiDAR data, Landsat, Sentinel-2, Sentinel-1, and PALSAR-2/PALSAR SAR data collected within New York’s Adirondack Park. Combining remote sensing data from multiple sources improved the model accuracy (RMSE: 52.14 Mg ha−1 and R2: 0.49). There was no significant difference among gradient boosting machine (GBM), random forest (RF), and extreme gradient boosting (XGBoost) models. In addition, pixel-based and object-based models were compared using the airborne LiDAR-derived AGB raster as a training/testing sample. The OBIA provided the best results with the RMSE of 33.77 Mg ha−1 and R2 of 0.81 for the combination of optical and SAR data in the GBM model.

Assessment of groundwater arsenic contamination using machine learning in Varanasi, Uttar Pradesh, India.

This paper presents a machine learning approach for classification of arsenic (As) levels as safe and unsafe in groundwater samples collected from the Indo-Gangetic region. As water is essential for sustaining life, heavy metals like arsenic pose a public health concern. In this study, various tree-based machine learning models namely Random Forest, Optimized Forest, CS Forest, SPAARC, and REP Tree algorithms have been applied to classify water samples. As per the guidelines of the World Health Organization (WHO), the arsenic concentration in water should not exceed 10 μg/L. The groundwater quality parameter was ranked using a classifier attribute evaluator for training and testing the models. Parameters obtained from the confusion matrix, such as accuracy, precision, recall, and FPR, were used to analyze the performance of models. Among all models, Optimized Forest outperforms other classifier as it has a high accuracy of 80.64%, a precision of 80.70%, recall of 97.87%, and a low FPR of 73.33%. The Optimized Forest model can be used to test new water samples for classification of arsenic in groundwater samples.

Protein pKa Prediction by Tree-Based Machine Learning.

Protonation states of ionizable protein residues modulate many essential biological processes. For correct modeling and understanding of these processes, it is crucial to accurately determine their pKa values. Here, we present four tree-based machine learning models for protein pKa prediction. The four models, Random Forest, Extra Trees, eXtreme Gradient Boosting (XGBoost), and Light Gradient Boosting Machine (LightGBM), were trained on three experimental PDB and pKa datasets, two of which included a notable portion of internal residues. We observed similar performance among the four machine learning algorithms. The best model trained on the largest dataset performs 37% better than the widely used empirical pKa prediction tool PROPKA and 15% better than the published result from the pKa prediction method DelPhiPKa. The overall root-mean-square error (RMSE) for this model is 0.69, with surface and buried RMSE values being 0.56 and 0.78, respectively, considering six residue types (Asp, Glu, His, Lys, Cys, and Tyr), and 0.63 when considering Asp, Glu, His, and Lys only. We provide pKa predictions for proteins in human proteome from the AlphaFold Protein Structure Database and observed that 1% of Asp/Glu/Lys residues have highly shifted pKa values close to the physiological pH.

Tree-based machine learning models for photovoltaic output power forecasting that consider photovoltaic panel soiling

ABSTRACT This article presents direct regression models for forecasting photovoltaic (PV) system output power based on machine learning (ML) methods, in which the Cleanness Index (CI) was introduced as an indicator of the PV panel soiling level. Three different ML methods based on decision trees were used: random forests (RF), extra trees (ET) and gradient boosting (GB). The research results showed that the introduction of the CI has significant potential to improve the forecasting of tree-based ML models. The mean average error of forecasts of the best model, which contained CI, and the model created by excluding CI from the set of input variables were 0.22% and 1.24%, respectively, as related to the nominal power of the PV panel. For the observed models, it was shown that the CI is the second-most important input variable and more significant than the PV panel temperature.

A Comparison of Infectious Disease Forecasting Methods across Locations, Diseases, and Time.

Accurate infectious disease forecasting can inform efforts to prevent outbreaks and mitigate adverse impacts. This study compares the performance of statistical, machine learning (ML), and deep learning (DL) approaches in forecasting infectious disease incidences across different countries and time intervals. We forecasted three diverse diseases: campylobacteriosis, typhoid, and Q-fever, using a wide variety of features (n = 46) from public datasets, e.g., landscape, climate, and socioeconomic factors. We compared autoregressive statistical models to two tree-based ML models (extreme gradient boosted trees [XGB] and random forest [RF]) and two DL models (multi-layer perceptron and encoder–decoder model). The disease models were trained on data from seven different countries at the region-level between 2009–2017. Forecasting performance of all models was assessed using mean absolute error, root mean square error, and Poisson deviance across Australia, Israel, and the United States for the months of January through August of 2018. The overall model results were compared across diseases as well as various data splits, including country, regions with highest and lowest cases, and the forecasted months out (i.e., nowcasting, short-term, and long-term forecasting). Overall, the XGB models performed the best for all diseases and, in general, tree-based ML models performed the best when looking at data splits. There were a few instances where the statistical or DL models had minutely smaller error metrics for specific subsets of typhoid, which is a disease with very low case counts. Feature importance per disease was measured by using four tree-based ML models (i.e., XGB and RF with and without region name as a feature). The most important feature groups included previous case counts, region name, population counts and density, mortality causes of neonatal to under 5 years of age, sanitation factors, and elevation. This study demonstrates the power of ML approaches to incorporate a wide range of factors to forecast various diseases, regardless of location, more accurately than traditional statistical approaches.

Interpretable tree-based ensemble model for predicting beach water quality

Tree-based machine learning models based on environmental features offer low-cost and timely solutions for predicting microbial fecal contamination in beach water to inform the public of the health risk. However, many of these models are black boxes that are difficult for humans to understand, which may cause severe consequences such as unexplained decisions and failure in accountability. To develop interpretable predictive models for beach water quality, we evaluate five tree-based models, namely classification tree, random forest, CatBoost, XGBoost, and LightGBM, and employ a state-of-the-art explanation method SHAP to explain the models. When tested on the Escherichia coli (E. coli) concentration data collected from three beach sites along Lake Erie shores, LightGBM, followed by XGBoost, achieves the highest averaged precision and recall scores. For all three sites, both models suggest lake turbidity as the most important predictor, and elucidate the crucial role of accurate local data of wave height and rainfall in the model development. Local SHAP values further reveal the robustness of the importance of lake turbidity as its SHAP value increases nearly monotonically with its value and is minimally affected by other environmental factors. Moreover, we found an intriguing interaction between lake turbidity and day-of-year. This work suggests that the combination of LightGBM and SHAP has a promising potential to develop interpretable models for predicting microbial water quality in freshwater lakes.