Ensemble Tree Method Research Articles

An interpretable ensemble trees method with joint analysis of static and dynamic features for myocardial infarction detection

Objective. In recent years, artificial intelligence-based electrocardiogram (ECG) methods have been massively applied to myocardial infarction (MI). However, the joint analysis of static and dynamic features to achieve accurate and interpretable MI detection has not been comprehensively addressed. Approach. This paper proposes a simplified ensemble tree method with a joint analysis of static and dynamic features to solve this issue for MI detection. Initially, the dynamic features are extracted by modeling the intrinsic dynamics of ECG via dynamic learning in addition to extracting classical static features. Secondly, a two-stage feature selection strategy is designed to identify a few significant features, which substitute the original variables that are employed in constructing the ensemble tree. This approach enhances the discriminative ability by selecting significant static and dynamic features. Subsequently, this paper presents an interpretable classification method named StackTree by introducing a stacked ensemble scheme to modify the ensemble tree simplification algorithm. The representative rules of the raw ensemble trees are selected as the intermediate training data that is used to retrain a decision tree with performance close to that of the source ensemble model. Using this scheme, the significant precision and interpretability of MI detection are thus comprehensively addressed. Main results. The effectiveness of our method in detecting MI is evaluated using the Physikalisch-Technische Bundesanstalt (PTB) and clinical database. The findings suggest that our algorithm outperforms the traditional methods based on a single type of feature. Additionally, it is comparable to the conventional random forest, achieving 97.1% accuracy under the inter-patient framework on the PTB database. Furthermore, feature subsets trained on PTB are validated using the clinical database, resulting in an accuracy of 84.5%. The chosen important features demonstrate that both static and dynamic information have crucial roles in MI detection. Crucially, the proposed method provides clear internal workings in an easy-to-understand visual manner.

Application of Ensemble Tree Algorithm for Installment Payment Arrears Prediction at Makmur Bersama Credit Union

<span lang="EN-US">One of the widely used machine learning techniques is the ensemble tree method, which is a combination of several classification trees where the final decision is based on the combined predictions of each tree. This approach produces better accuracy than a single classification tree. Two common methods used in the ensemble tree technique are boosting and bagging. This research will predict the status of installment payments at CU Makmur Bersama Credit Union. The method used is the bagging tree method, namely random forest and boosting, namely AdaBoost. To get optimal results, hyperparameter tuning is also carried out. The results showed that the boosting and bagging ensemble tree methods were able to handle the classification of cooperative loan installment payment status better than the distance approach, namely kNN (single classification). The performance of the boosting ensemble tree with the AdaBoost model has an accuracy of 72.89% better than the bagging ensemble tree with the random forest model whose accuracy is 72.08%.</span>

Using AutoML and generative AI to predict the type of wildfire propagation in Canadian conifer forests

Accurate prediction of wildfire behaviour is critical for safe and effective fire management and suppression. This study focused on developing and evaluating machine learning (ML) models based on a dataset collected of outdoor experimental fires in Canadian conifer forests. Binary classification models (surface fire vs. crown fire) and multi-class fire type models (surface, passive crown, active crown) were created using ensemble tree methods (Random Forest, XGBoost) and automated ML (AutoGluon, TabPFN). Generative adversarial networks (GAN) were used to generate synthetic data to overcome imbalances in the type of fire distribution. The results show automated ML methods applied to the binary problem to perform best of all tested methods, with an overall accuracy of 91%. TabPFN had the highest accuracy for the multi-class problem, with the use of GAN improving model fit. Results show overfitting issues with some of the ML models, highlighting the need of independent evaluation when ML models are developed. The TabPFN models have potential for application in supporting fire management, namely in fuel management and identifying situations with high fire spread and intensity potential that can impact firefighter safety.

A comparative study on the most effective machine learning model for blast loading prediction: From GBDT to Transformer

In this paper, we present a rigorous comparative study to assess and identify the most effective machine learning model for blast loading prediction. Blast loads are known to produce catastrophic effects including structural collapse and personnel fatality. Accurate and efficient prediction of these extreme loads using empirical methods and numerical solvers remains a challenging problem. Machine learning provides a promising alternative solution, which has been increasingly used in various engineering applications. However, there is seldom any analysis or justification of the selection of machine learning method that would lead to the best performance for such applications. For example, most existing machine learning-based approaches for blast loading prediction utilise the classic multi-layer perceptron (MLP) network with no justifications of their suitability and efficiency nor attempts of leveraging other state-of-the-art neural network architectures. In this study, four well-known machine learning models, including one ensemble tree method and three neural networks of different types, are investigated to demonstrate the effectiveness of different machine learning methods for blast loading prediction. It is showcased using BLEVE (boiling liquid expanding vapour explosion) pressure prediction that the Transformer model achieves the best performance, reaching a relative error of 3.5% and R2 0.997 that outperforms the existing MLP approach (relative error 6.0%, R2 0.985) with a clear margin. This study shows that the Transformer network is an effective tool for prediction of blast loading from BLEVE as well as other explosion sources.

Using Tree-Based Machine Learning for Health Studies: Literature Review and Case Series.

Tree-based machine learning methods have gained traction in the statistical and data science fields. They have been shown to provide better solutions to various research questions than traditional analysis approaches. To encourage the uptake of tree-based methods in health research, we review the methodological fundamentals of three key tree-based machine learning methods: random forests, extreme gradient boosting and Bayesian additive regression trees. We further conduct a series of case studies to illustrate how these methods can be properly used to solve important health research problems in four domains: variable selection, estimation of causal effects, propensity score weighting and missing data. We exposit that the central idea of using ensemble tree methods for these research questions is accurate prediction via flexible modeling. We applied ensemble trees methods to select important predictors for the presence of postoperative respiratory complication among early stage lung cancer patients with resectable tumors. We then demonstrated how to use these methods to estimate the causal effects of popular surgical approaches on postoperative respiratory complications among lung cancer patients. Using the same data, we further implemented the methods to accurately estimate the inverse probability weights for a propensity score analysis of the comparative effectiveness of the surgical approaches. Finally, we demonstrated how random forests can be used to impute missing data using the Study of Women's Health Across the Nation data set. To conclude, the tree-based methods are a flexible tool and should be properly used for health investigations.

Quantitative Structure-Activity Relationship (QSAR) Study Predicts Small-Molecule Binding to RNA Structure.

The diversity of RNA structural elements and their documented role in human diseases make RNA an attractive therapeutic target. However, progress in drug discovery and development has been hindered by challenges in the determination of high-resolution RNA structures and a limited understanding of the parameters that drive RNA recognition by small molecules, including a lack of validated quantitative structure–activity relationships (QSARs). Herein, we develop QSAR models that quantitatively predict both thermodynamic- and kinetic-based binding parameters of small molecules and the HIV-1 transactivation response (TAR) RNA model system. Small molecules bearing diverse scaffolds were screened against TAR using surface plasmon resonance. Multiple linear regression (MLR) combined with feature selection afforded robust models that allowed direct interpretation of the properties critical for both binding strength and kinetic rate constants. These models were validated with new molecules, and their accurate performance was confirmed via comparison to ensemble tree methods, supporting the general applicability of this platform.

Accurate identification of Parkinson’s disease by distinctive features and ensemble decision trees

Parkinson’s disease (PD) is a progressive neurological disorder that primarily leads to a series of motor impairments. Therefore, human gait patterns and information obtained from various sensors are employed to extract distinctive features for recognizing the difference between healthy controls and PD patients. However, improper analysis of these gait symptoms may mislead the diagnosis of PD due to gradually progressive characteristics of gait disorders. Moreover, individual differences of measuring signals are often preferable to the gait intrinsic changes induced by PD. To deal with those issues, the mean, coefficient variance (CV), and asymmetry index (AI) of temporal, VGRF/BW based, and ED-based features are extracted and compared by the violin plot and Mann-Whitney U-Test to find the distinctive features and discernible changes of the PD gait. Moreover, ensemble decision trees is proposed for accurate PD diagnosis. The ensemble decision trees with features from time, VGRF/BW, and ED are tested and evaluated by the prediction accuracy. Results show that based on the mean, CV, and AI of VGRF/BW at both posterior, inside and outside heel, inside and outside arch, inside and outside sole, toe, and the total force of left and right, the proposed ensemble tree method achieves a mean accuracy of 99.52% with a standard deviation of 0.10%. The distinctive features and accurate diagnosis will be helpful for the home-based and continuous monitoring to improve treatment and therapy of PD patients.

Student Behavior Analysis to Predict Learning Styles Based Felder Silverman Model Using Ensemble Tree Method

Learning styles are very important to know so that students can learn effectively. By understanding the learning style, students will learn about their needs in the learning process. One of the famous learning management systems is called Moodle. Moodle can catch student experiences and behaviors while learning and store all student activities in the Moodle Log. There is a fundamental issue in e-learning where not all students have the same degree of comprehension. Therefore, in some cases of learning in E-Learning, students tend to leave the classroom and lack activeness in the classroom. In order to solve these problems, we have to know students' preferences in the learning process by understanding each student's learning style. To find out the appropriate student learning style, it is necessary to analyze student behavior based on the frequency of visits when accessing Moodle E-learning and fill out the Index Learning Style (ILS) questionnaire. The Felder Silverman model's learning style classifies it into four dimensions: Input, Processing, Perception, and Understanding. We propose a learning style prediction model using the Ensemble Tree method, namely Bagging and Boosting-Gradient Boosted Tree. Afterwards, we evaluate the classification results using Stratified Cross Validation and measure the performance using accuracy. The results showed that the Ensemble Tree method's classification efficiency has higher accuracy than a single tree classification model.

Assessing the predictive capability of ensemble tree methods for landslide susceptibility mapping using XGBoost, gradient boosting machine, and random forest

Decision tree-based classifier ensemble methods are a machine learning (ML) technique that combines several tree models to produce an effective or optimum predictive model, and that allows well-predictive performance especially compared to a single model. Thus, selecting a proper ML algorithm help us to understand possible future occurrences by analyzing the past more accurate. The main purpose of this study is to produce landslide susceptibility map of the Ayancik district of Sinop province, situated in the Black Sea region of Turkey using three featured regression tree-based ensemble methods including gradient boosting machines (GBM), extreme gradient boosting (XGBoost), and random forest (RF). Fifteen landslide causative factors and 105 landslide locations occurred in the region were used. The landslide inventory map was randomly divided into training (70%) and testing (30%) dataset to construct the RF, XGBoost and GBM prediction models. Symmetrical uncertainty measure was utilized to determine the most important causative factors, and then the selected features were used to construct susceptibility prediction models. The performance of the ensemble models was validated using different accuracy metrics including Area under the curve (AUC), overall accuracy (OA), Root mean square error (RMSE), and Kappa coefficient. Also, the Wilcoxon signed-rank test was used to assess differences between optimum models. The accuracy results showed that the model of XgBoost_Opt model (the model created by optimum factor combination) has the highest prediction capability (OA = 0.8501 and AUC = 0.8976), followed by the RF_opt (OA = 0.8336 and AUC = 0.8860) and GBM_Opt (OA = 0.8244 and AUC = 0.8796). When the Wilcoxon sign-rank test results were analyzed, XgBoost_Opt model, which is the best subset combinations, were confirmed to be statistically significant considering other models. The results showed that, the XGBoost method according to optimum model achieved lower prediction error and higher accuracy results than the other ensemble methods.

Nowcasting of the probability of accumulated precipitation based on the radar echo extrapolation

The study presents a new method nowcasting precipitation called the Ensemble Tree Method (ETM), which gives probability forecast of accumulated precipitation based on the extrapolation of radar reflectivity. ETM combines a tree model with a Bootstrap technique. It forecasts the probability that the hourly accumulated precipitation exceeds a given threshold for cells of 3 by 3 km size.ETM was tested using radar reflectivity data from July 2012 in a domain of 489 km by 291 km covering the Czech Republic (Central Europe). While forecasting, we considered a lead time of up to 180 min having a time step of 30 min and four precipitation thresholds (0.1, 1.0, 5.0, and 10.0 mm). ETM provided us forecasts of the probability of exceeding an hourly precipitation threshold from 0 to 60 min, 30 to 90 min, …, and 120 to 180 min. The performance of ETM was assessed using a skill score derived from the mean-square-error, and was compared with the performance of forecasts based on a logistic regression that was used as reference forecast. We demonstrated that the prediction of ETM is better than that of the reference forecast. The main advantage of ETM is that the ETM reflects the uncertainty of forecast better as compared to the overconfident reference forecasts, which is particularly true for the higher precipitation thresholds. Thus, despite low predicted probabilities, the forecasts given by ETM seem more realistic.

Classification and regression using augmented trees

In this work, we present an algorithm for regression and classification tasks on big datasets using augmented tree models. Partitioning a big dataset using a tree model permits us to apply a divide and conquer strategy to classification and regression tasks. Experiments conducted as part of this study illustrate that such an approach has an important benefit. Methods associated with good accuracies on learning tasks on big datasets such as ensemble tree methods or neural networks produce models that are not interpretable. The models produced by the proposed algorithm are interpretable while being as accurate as ensemble methods such as random forests or gradient boosted trees. Model interpretation can be performed at coarse and fine granularity. This permits us to extract insights that characterize the entire dataset or a particular subset of the data. Models that are accurate and interpretable are highly desirable in many application settings. The partitions created by the algorithm also permit a divide and conquer approach to model analysis. Analysis of performance by partition helped identify problems such as possible data errors and model overfitting.

Potential of ensemble tree methods for early-season prediction of winter wheat yield from short time series of remotely sensed normalized difference vegetation index andin situmeteorological data

We aimed at analyzing the potential of two ensemble tree machine learning methods—boosted regression trees and random forests—for (early) prediction of winter wheat yield from short time series of remotely sensed vegetation indices at low spatial resolution and of in situ meteorological data in combination with annual fertilization levels. The study area was the Huaibei Plain in eastern China, and all models were calibrated and validated for five separate prefectures. To this end, a cross-validation process was developed that integrates model meta-parameterization and simple forward feature selection. We found that the resulting models deliver early estimates that are accurate enough to support decision making in the agricultural sector and to allow their operational use for yield forecasting. To attain maximum prediction accuracy, incorporating predictors from the end of the growing season is, however, recommended.

Random Survival Forests

Random Survival Forests