Tree-based Learning Algorithms Research Articles

Predictive Models for Seismic Source Parameters Based on Machine Learning and General Orthogonal Regression Approaches

ABSTRACT Two sets of predictive models are developed based on the machine learning (ML) and general orthogonal regression (GOR) approaches for predicting the seismic source parameters including rupture width, rupture length, rupture area, and two slip parameters (i.e., the average and maximum slips of rupture surface). The predictive models are developed based on a compiled catalog consisting of 1190 sets of estimated source parameters. First, the Light Gradient Boosting Machine (LightGBM), which is a gradient boosting framework that uses tree-based learning algorithms, is utilized to develop the ML-based predictive models by employing five predictor variables consisting of moment magnitude (Mw), hypocenter depth, dip angle, fault-type, and subduction indicators. It is found that the developed ML-based models exhibit good performance in terms of predictive efficiency and generalization. Second, multiple source-scaling models are developed for predicting the source parameters based on the GOR approach, in which each functional form has one predictor variable only, that is, Mw. The performance of the GOR-based models is compared with existing source-scaling relationships. Both sets of the models developed are applicable in estimating the five source parameters in earthquake engineering-related applications.

Forgetful Forests: Data Structures for Machine Learning on Streaming Data under Concept Drift

Database and data structure research can improve machine learning performance in many ways. One way is to design better algorithms on data structures. This paper combines the use of incremental computation as well as sequential and probabilistic filtering to enable “forgetful” tree-based learning algorithms to cope with streaming data that suffers from concept drift. (Concept drift occurs when the functional mapping from input to classification changes over time). The forgetful algorithms described in this paper achieve high performance while maintaining high quality predictions on streaming data. Specifically, the algorithms are up to 24 times faster than state-of-the-art incremental algorithms with, at most, a 2% loss of accuracy, or are at least twice faster without any loss of accuracy. This makes such structures suitable for high volume streaming applications.

Classification models for impact damage of fiber reinforced concrete panels using Tree-based learning algorithms

This study aims to develop the machine learning models for classification of the local damage levels of FRC panels subjected to missile impact load using the tree-based algorithms and ensembles. Six different algorithms, including Decision Tree, Random Forest, Bagging, AdaBoost, XGBoost, and CatBoost were trained and evaluated based on a dataset collected from 176 experiments of FRC panels under missile impact, which consists of 15 input parameters of geometries, materials, and boundary conditions and one output parameter of local damage level of FRC panels. The Bayesian Optimization algorithm and k-fold cross validation were also utilized to achieve higher accuracy in prediction ability of the models. The obtained results showed that the proposed models can predict the local damage of FRC panels subjected to the missile impact load with acceptable accuracy. The models using ensemble methods have better performance than single estimator model in prediction and each model using ensemble method has its own strength and is suitable for different criteria when classifying. For imbalanced dataset, Random Forest can be chosen as the most suitable classification model for the dataset in this study.

Digital mapping for soil texture class prediction in northwestern Türkiye by different machine learning algorithms

Soil texture classes (STCs) influence the physical, chemical and biological properties of the soil, and accurate spatial predictions of STCs are essential for agro-ecological modeling. The purpose of this study was to assess the capabilities of environmental covariates derived from Landsat 8 OLI Science products and a digital elevation model (DEM), as well as three machine learning methods, to provide an accurate and reliable classification of soil texture classes. Estimation efficiency of soil texture classes was investigated using Decision tree (DT), Random forest (RF), and Support vector machine (SVM) algorithms in an area with Fluvisols and Vertisols as predominant reference soil groups in Northwestern Türkiye (Lake of Manyas). The models were validated using the leave-one-out, cross-validation technique. The best of the three machine learning models for soil texture classification was RF, with an overall accuracy of 0.63 and a kappa index value of 0.14, according to the accuracy evaluation. The RF algorithm generated a map whose findings were more consistent with the real environment using the confusion index (RF:0.30) and abundance index (RF:1.0) values as uncertainty criteria. The most important predictor of soil texture classes was the topographic wetness index (mean decrease accuracy: 3%) for tree-based learning algorithms, followed by other indexes based on satellite data. These findings will contribute to support sustainable soil management techniques in a location with a large degree of topsoil texture diversity.

A case study for assessing the utility of a decision tree based learning algorithm in mental health inpatient care quality management

IntroductionThere is limited knowledge about the potential role of machine learning (ML) in quality improvement of psychiatric care.ObjectivesOur case study was to determine whether ML decision trees used on patient databases are suitable for focussing on specific patient population samples of mental healthcare quality audits. Populations were identified by patient and care provider variables, and the time of treatment. Outcomes were defined as hospital mortality, over-long hospitalization (over average +1SD or +2SD); and short hospitalization (under average -1SD; under 3 days).MethodsWe conducted a Split Train Test in Python for our outcomes on national mental health inpatient turnover data (2010 through 2018 for training and 2019 for testing). A well-fitting decision tree had the area under the curve (AUC) of the receiver operating characteristic (ROC) >= 0.7, and specificity >= 0.9. Performing qualitative analyses of decision trees, we rejected the ones with little clinical relevance.ResultsDecision trees fit well (AUC = 0.7 to 0.9; specificity = 0.7 to 1.0; sensitivity = 0 to 0.69). For hospital death cases, the decision tree had AUC = 0.86, no difference after controlling for the types of hospital units, and was clinically relevant. Models predicting over-long hospitalization fit well (AUC=0,9); however, controlling for care pathways, good fit and sensitivity both vanished. No valid models emerged for undertime discharges. The decision trees revealed unique combinations of variables.ConclusionsOur ML decision trees used on healthcare databases proved promising for focussing quality audit efforts. Narrative analysis for the clinical contexts of the decision trees is indispensable.DisclosureNo significant relationships.

Impact of geophysical and anthropogenic factors on wildfire size: a spatiotemporal data-driven risk assessment approach using statistical learning

Wildfire spread is a stochastic phenomenon driven by a multitude of geophysical and anthropogenic factors. In this study, we propose a spatiotemporal data-driven risk assessment framework to understand the effect of various geophysical/anthropogenic factors on wildfire size, leveraging a systematic machine learning approach. We apply this framework in the state of California–the most vulnerable US state to wildfires. Using county-level annual wildfire data from 2001–2015, and various geophysical (e.g., land cover, wind, surface temperature) and anthropogenic features (e.g., population density, housing type), we trained, tested, and validated a suite of ensemble tree-based learning algorithms to identify and evaluate the key factors associated with wildfire size. The Extreme Gradient Boosting (XGBoost) algorithm outperformed all the other models in terms of generalization performance, categorization of important features, and risk performance. We found that standard deviations of meteorological variables with long-tailed distributions play a key role in predicting wildfire size. Specifically, the top ten factors associated with high risk of larger wildfires include larger standard deviations of surface temperature and vapor pressure deficit, higher wind gust, more grassy and barren land covers, lower night-time boundary layer height and higher population density. Our proposed risk assessment framework will help federal/state decision-makers to adequately plan for wildfire risk mitigation and resource allocation strategies.

Diagnosis of Diabetes Mellitus Using Gradient Boosting Machine (LightGBM).

Diabetes mellitus (DM) is a severe chronic disease that affects human health and has a high prevalence worldwide. Research has shown that half of the diabetic people throughout the world are unaware that they have DM and its complications are increasing, which presents new research challenges and opportunities. In this paper, we propose a preemptive diagnosis method for diabetes mellitus (DM) to assist or complement the early recognition of the disease in countries with low medical expert densities. Diabetes data are collected from the Zewditu Memorial Hospital (ZMHDD) in Addis Ababa, Ethiopia. Light Gradient Boosting Machine (LightGBM) is one of the most recent successful research findings for the gradient boosting framework that uses tree-based learning algorithms. It has low computational complexity and, therefore, is suited for applications in limited capacity regions such as Ethiopia. Thus, in this study, we apply the principle of LightGBM to develop an accurate model for the diagnosis of diabetes. The experimental results show that the prepared diabetes dataset is informative to predict the condition of diabetes mellitus. With accuracy, AUC, sensitivity, and specificity of 98.1%, 98.1%, 99.9%, and 96.3%, respectively, the LightGBM model outperformed KNN, SVM, NB, Bagging, RF, and XGBoost in the case of the ZMHDD dataset.

The Predictability of Tree-based Machine Learning Algorithms in the Big Data Context

This research work is concerned with the predictability of ensemble and singular tree-based machine learning algorithms during the recession and prosperity of the two companies listed in the Tehran Stock Exchange in the context of big data. In this regard, the main issue is that economic managers and the academic community require predicting models with more accuracy and reduced execution time; moreover, the prediction of the companies recession in the stock market is highly significant. Machine learning algorithms must be able to appropriately predict the stock return sign during the market downturn and boom days. Addressing the stated challenge will upgrade the quality of stock purchases and, subsequently, will increase profitability. In this article, the proposed solution relies on the utilization of tree-based machine learning algorithms in the context of big data. The proposed solution exploits the decision tree algorithm, which is a traditional and singular tree-based learning algorithm. Furthermore, two modern and ensemble tree-based learning algorithms, random forest and gradient boosted tree, has been utilized for predicting the stock return sign during recession and prosperity. The mentioned cases were implemented by applying the machine learning tools in python programming language and PYSPARK library that is used explicitly for the big data context. The utilized research data of the current study are the shares information of two companies of the Tehran Stock Exchange. The obtained results reveal that the applied ensemble learning algorithms have performed better than the singular learning algorithms. Additionally, adding 23 technical features to the initial data and subsequent applying of the PCA feature reduction method have demonstrated the best performance among other modes. In the meantime, it has been concluded that the initial data do not possess the proper resolution or generalizability, either during prosperity or recession.

A novel learning algorithm for Büchi automata based on family of DFAs and classification trees

In this paper, we propose a novel algorithm to learn a Büchi automaton from a teacher who knows an ω-regular language. The learned Büchi automaton can be a nondeterministic Büchi automaton or a limit deterministic Büchi automaton. The learning algorithm is based on learning a formalism called family of DFAs (FDFAs) recently proposed by Angluin and Fisman. The main catch is that we use a classification tree structure instead of the standard observation table structure. The worst case storage space required by our algorithm is quadratically better than that required by the table-based algorithm proposed by Angluin and Fisman. We implement the proposed learning algorithms in the learning library ROLL (Regular Omega Language Learning), which also consists of other complete ω-regular learning algorithms available in the literature. Experimental results show that our tree-based learning algorithms have the best performance among others regarding the number of solved learning tasks.

Correcting bias of satellite rainfall data using physical empirical model

The provision of high resolution near real-time rainfall data has made satellite rainfall products very potential for monitoring hydrological hazards. However, a major challenge in their direct-use can be problematic due to measurement error. In this study, an attempt was made to correct the bias of Global Satellite Mapping of Precipitation near-real-time (GSMaP_NRT) product. Physical factors, including topography, season, windspeed and cloud types were accounted for correcting bias. Peninsular Malaysia was used as the case study area. Gridded rainfall, developed from 80 gauges for the period 2000–2018, was used along with physical factors in a two-stage procedure. The model consisted of a classifier to categorise rainfall of different intensity and regression models to predict rainfall amount of different intensity class. An ensemble tree-based learning algorithm, called random forest, was used for classification and regression. The results revealed a big improvement of near-real-time GSMaP_NRT product after bias correction (GSMaP_BC) compared to the gauge corrected version (GSMaP_GC). Accuracy evaluation for complete timeseries indicated about 110% reduction of normalized root-mean-square error (NRMSE) in GSMaP_BC (0.8) compared to GSMaP_NRT (1.7) and GSMaP_GC (1.75). On the other hand, the bias of GSMaP_BC became nearly zero (0.3) compared to 2.1 and − 3.1 for GSMaP_NRT and GSMaP_GC products. The spatial correlation of GSMaP_BC was >0.7 with observed rainfall data for all months compared to 0.2–0.78 for GSMaP_NRT and GSMaP_GC, indicating capability of GSMaP_BC to replicate spatial pattern of rainfall. The bias-corrected near-real-time GSMaP data can be used for monitoring and forecasting floods and hydrological phenomena in the absence of dense rain-gauge network in areas, frequently experience hydro-meteorological hazards.

Long-Term Wind Power Forecasting Using Tree-Based Learning Algorithms

The intermittent and uncertain nature of wind places a premium on accurate wind power forecasting for the reliable and efficient operation of power grids with large-scale wind power penetration. Herein, six-month-ahead wind power forecasting models were developed using tree-based learning algorithms. Three models were developed to investigate the impact of input data on forecasting accuracy. The first model was trained with the average and standard deviation of wind speed values measured at a height of 40 m with a 10-min sampling time. To evaluate the impact of sampling time on model performance, a second model was trained with wind speed values measured at a height of 40 m with 1-h, 12-h, and 24-h sampling times. To assess the effect of measuring height on model accuracy, the third model was trained with wind speed values measured at 40 m extrapolated from values measured at heights of 30 m and 10 m. Experiments revealed that using longer time intervals and height extrapolation leads to considerable accuracy degradation in forecasted models. Finally, to study the generalization ability of the forecasted models, they were tested against wind data measured at heights and locations different from what the models had been trained with. Simulation results substantiated that tree-based learning algorithms can be successfully adopted not only for long-term wind power forecasting, but for potential wind power forecasting at different heights and geographical locations.

FPGA Acceleration of Tree-based Learning Algorithms

FPGA Acceleration of Tree-based Learning Algorithms

Prediction of Material Removal Rate for Chemical Mechanical Planarization Using Decision Tree-Based Ensemble Learning

Chemical mechanical planarization (CMP) has been widely used in the semiconductor industry to create planar surfaces with a combination of chemical and mechanical forces. A CMP process is very complex because several chemical and mechanical phenomena (e.g., surface kinetics, electrochemical interfaces, contact mechanics, stress mechanics, hydrodynamics, and tribochemistry) are involved. Predicting the material removal rate (MRR) in a CMP process with sufficient accuracy is essential to achieving uniform surface finish. While physics-based methods have been introduced to predict MRRs, little research has been reported on monitoring and predictive modeling of the MRR in CMP. This paper presents a novel decision tree-based ensemble learning algorithm that can train the predictive model of the MRR. The stacking technique is used to combine three decision tree-based learning algorithms, including the random forests (RF), gradient boosting trees (GBT), and extremely randomized trees (ERT), via a meta-regressor. The proposed method is demonstrated on the data collected from a CMP tool that removes material from the surface of wafers. Experimental results have shown that the decision tree-based ensemble learning algorithm using stacking can predict the MRR in the CMP process with very high accuracy.

Data-driven Approach to Material Removal Rate Prediction in Chemical Mechanical Polishing

Chemical mechanical polishing (CMP) has been widely used in the semiconductor sector for creating planar surfaces with the combination of chemical and mechanical forces. CMP is very complex because several chemical and mechanical phenomena (e.g., surface kinetics, contact mechanics, stress mechanics, and tribochemistry) are involved. Due to the complexity of the CMP process, it is very challenging to predict material removal rate (MRR) with sufficient accuracy. While physics-based methods have been introduced to predict MRR, little research has been reported on data-driven predictive modeling of MRR in the CMP process. This paper presents a novel decision tree-based ensemble learning algorithm that trains a predictive model of MRR on condition monitoring data. A stacking technique is used to combine three decision tree-based learning algorithms, including the random forests (RF), gradient boosting trees (GBT), and extremely randomized trees (ERT). The proposed method is demonstrated on the data collected from a wafer CMP tool that removes material from the surface of the wafer. Experimental results have shown that the decision tree-based ensemble learning algorithm can predict MRR in the CMP process with very high accuracy.

Prediction of concrete compressive strength using non-destructive test results

Concrete which is a composite material is one of the most important construction materials. Compressive strength is a commonly used parameter for the assessment of concrete quality. Accurate prediction of concrete compressive strength is an important issue. In this study, we utilized an experimental procedure for the assessment of concrete quality. Firstly, the concrete mix was prepared according to C 20 type concrete, and slump of fresh concrete was about 20 cm. After the placement of fresh concrete to formworks, compaction was achieved using a vibrating screed. After 28 day period, a total of 100 core samples having 75 mm diameter were extracted. On the core samples pulse velocity determination tests and compressive strength tests were performed. Besides, Windsor probe penetration tests and Schmidt hammer tests were also performed. After setting up the data set, twelve artificial intelligence (AI) models compared for predicting the concrete compressive strength. These models can be divided into three categories (i) Functions (i.e., Linear Regression, Simple Linear Regression, Multilayer Perceptron, Support Vector Regression), (ii) Lazy-Learning Algorithms (i.e., IBk Linear NN Search, KStar, Locally Weighted Learning) (iii) Tree-Based Learning Algorithms (i.e., Decision Stump, Model Trees Regression, Random Forest, Random Tree, Reduced Error Pruning Tree). Four evaluation processes, four validation implements (i.e., 10-fold cross validation, 5-fold cross validation, 10% split sample validation & 20% split sample validation) are used to examine the performance of predictive models. This study shows that machine learning regression techniques are promising tools for predicting compressive strength of concrete.

Head and neck target delineation using a novel PET automatic segmentation algorithm

PurposeTo evaluate the feasibility and impact of using a novel advanced PET auto-segmentation method in Head and Neck (H&N) radiotherapy treatment (RT) planning. MethodsATLAAS, Automatic decision Tree-based Learning Algorithm for Advanced Segmentation, previously developed and validated on pre-clinical data, was applied to 18F-FDG-PET/CT scans of 20 H&N patients undergoing Intensity Modulated Radiation Therapy. Primary Gross Tumour Volumes (GTVs) manually delineated on CT/MRI scans (GTVpCT/MRI), together with ATLAAS-generated contours (GTVpATLAAS) were used to derive the RT planning GTV (GTVpfinal). ATLAAS outlines were compared to CT/MRI and final GTVs qualitatively and quantitatively using a conformity metric. ResultsThe ATLAAS contours were found to be reliable and useful. The volume of GTVpATLAAS was smaller than GTVpCT/MRI in 70% of the cases, with an average conformity index of 0.70. The information provided by ATLAAS was used to grow the GTVpCT/MRI in 10 cases (up to 10.6mL) and to shrink the GTVpCT/MRI in 7 cases (up to 12.3mL). ATLAAS provided complementary information to CT/MRI and GTVpATLAAS contributed to up to 33% of the final GTV volume across the patient cohort. ConclusionsATLAAS can deliver operator independent PET segmentation to augment clinical outlining using CT and MRI and could have utility in future clinical studies.

ATLAAS: an automatic decision tree-based learning algorithm for advanced image segmentation in positron emission tomography

Accurate and reliable tumour delineation on positron emission tomography (PET) is crucial for radiotherapy treatment planning. PET automatic segmentation (PET-AS) eliminates intra- and interobserver variability, but there is currently no consensus on the optimal method to use, as different algorithms appear to perform better for different types of tumours. This work aimed to develop a predictive segmentation model, trained to automatically select and apply the best PET-AS method, according to the tumour characteristics.ATLAAS, the automatic decision tree-based learning algorithm for advanced segmentation is based on supervised machine learning using decision trees. The model includes nine PET-AS methods and was trained on a 100 PET scans with known true contour. A decision tree was built for each PET-AS algorithm to predict its accuracy, quantified using the Dice similarity coefficient (DSC), according to the tumour volume, tumour peak to background SUV ratio and a regional texture metric. The performance of ATLAAS was evaluated for 85 PET scans obtained from fillable and printed subresolution sandwich phantoms.ATLAAS showed excellent accuracy across a wide range of phantom data and predicted the best or near-best segmentation algorithm in 93% of cases. ATLAAS outperformed all single PET-AS methods on fillable phantom data with a DSC of 0.881, while the DSC for H&N phantom data was 0.819. DSCs higher than 0.650 were achieved in all cases.ATLAAS is an advanced automatic image segmentation algorithm based on decision tree predictive modelling, which can be trained on images with known true contour, to predict the best PET-AS method when the true contour is unknown. ATLAAS provides robust and accurate image segmentation with potential applications to radiation oncology.

An Empirical Comparison of Tree-Based Learning Algorithms: An Egyptian Rice Diseases Classification Case Study

Applications of learning algorithms in knowledge discovery are promising and relevant area of research. The classification algorithms of data mining have been successfully applied in the recent years to predict Egyptian rice diseases. Various classification algorithms can be applied on such data to devise methods that can predict the occurrence of diseases. However, the accuracy of such techniques differ according to the learning and classification rule used. Identifying the best classification algorithm among all available is a challenging task. In this study, a comprehensive comparative analysis of a tree-based different classification algorithms and their performance has been evaluated by using Egyptian rice diseases data set. The experimental results demonstrate that the performance of each classifier and the results indicate that the decision tree gave the best results.

Application of local linear neuro-fuzzy model in estimating reservoir water saturation from well logs

The objective of the present study is to show an application of local linear neuro-fuzzy (LLNF) model in estimating reservoir water saturation (Sw) from well logs in a carbonate reservoir. A locally linear model tree (LOLIMOT), as an incremental tree-based learning algorithm, is actively used for optimizing the rule premise parameters. To show the relative strength of the proposed method in estimating Sw, the results obtained from LLNF are compared with three conventional equations. Moreover, the results are also evaluated on the basis of two criteria including “determination coefficient (R2)” and “root mean square error (RMSE).” The R2 for the estimated Sw is 0.93 using the LLNF model, while it is approximated about 0.81 by the “Dual-water (Clavier et al. 1984) equation,” as the best-fitted empirical equation. Also, the RMSE obtained is 0.023 by LLNF model, while it is 0.042 by Dual-water.

Multi-criteria route planning based on a driver’s preferences in multi-criteria route selection

In this study, some different approaches were designed, implemented, and evaluated to perform multi-criteria route planning by considering a driver’s preferences in multi-criteria route selection. At first, by using a designed neuro-fuzzy toolbox, the driver’s preferences in multi-criteria route selection such as the preferred criteria in route selection, the number of route-rating classes, and the routes with the same rate were received. Next, to learn the driver’s preferences in multi-criteria route selection and to classify any route based on these preferences, a methodology was proposed using a locally linear neuro-fuzzy model (LLNFM) trained with an incremental tree based learning algorithm. In this regard, the proposed LLNFM-based methodology reached better results for running-times, as well as root mean square error (RMSE) estimations in learning and testing processes of training/checking data-set in comparison with those of the proposed adaptive neuro-fuzzy inference system (ANFIS) based methodology. Finally, the trained LLNFM-based methodology was utilized to plan and predict a driver’s preferred routes by classifying Pareto-optimal routes obtained by running the modified invasive weed optimization (IWO) algorithm between an origin and a destination of a real urban transportation network based on the driver’s preferences in multi-criteria route selection.