Bagging Of Decision Trees Research Articles

A new intuitionistic fuzzy time series method based on the bagging of decision trees and principal component analysis

A new intuitionistic fuzzy time series method based on the bagging of decision trees and principal component analysis

Prediction models for marshall mix parameters using bio-inspired genetic programming and deep machine learning approaches: A comparative study

This research study utilizes four machine learning techniques, i.e., Multi Expression programming (MEP), Artificial Neural Network (ANN), Adaptive Neuro-Fuzzy Inference System (ANFIS), and Ensemble Decision Tree Bagging (DT-Bagging) for the development of new and advanced models for prediction of Marshall Stability (MS), and Marshall Flow (MF) of asphalt mixes. A comprehensive and detailed database of 343 data points was established for both MS and MF. The predicting variables were chosen among the four most influential, and easy-to-determine parameters. The models were trained, tested, validated, and the outcomes of the newly developed models were compared with actual outcomes. The root squared error (RSE), Nash-Sutcliffe efficiency (NSE), mean absolute error (MAE), root mean square error (RMSE), relative root mean square error (RRMSE), regression coefficient (R2), and correlation coefficient (R), were all used to evaluate the performance of models. The sensitivity analysis (SA) revealed that in the case of MS, the rising order of input significance was bulk specific gravity of compacted aggregate, Gmb (38.56 %) > Percentage of Aggregates, Ps (19.84 %) > Bulk Specific Gravity of Aggregate, Gsb (19.43 %) > maximum specific gravity paving mix, Gmm (7.62 %), while in case of MF the order followed was: Ps (36.93 %) > Gsb (14.11 %) > Gmb (10.85 %) > Gmm (10.19 %). The outcomes of parametric analysis (PA) consistency of results in relation to previous research findings. The DT-Bagging model outperformed all other models with values of 0.971 and 0.980 (R), 16.88 and 0.24 (MAE), 28.27 and 0.36 (RMSE), 0.069 and 0.041 (RSE), 0.020 and 0.032 (RRMSE), 0.010 and 0.016 (PI), 0.931 and 0.959 (NSE), for MS and MF, respectively. The results of the comparison analysis showed that ANN, ANFIS, MEP, and DT-Bagging are all effective and reliable approaches for the estimation of MS and MF. The MEP-derived mathematical expressions represent the novelty of MEP and are relatively simple and reliable. Roverall values for MS and MF were in the order of DT-Bagging >MEP >ANFIS >ANN with all values exceeding the permitted range of 0.80 for both MS and MF. Hence, all the modeling approaches showed higher performance, possessed high generalization and predication capabilities, and assess the relative significance of input parameters in the prediction of MS and MF. Hence, the findings of this research study would assist in the safer, faster, and sustainable prediction of MS and MF, from the standpoint of resources and time required to perform the Marshall tests.

Enhancing Heart Disease Prediction Using Ensemble Techniques

Background: Cardiovascular diseases are recognized generally to be among the number one illnesscausing death across the globe. Predicting heart disease using a computer-aided technique makes it easier for medical practitioners to diagnose and thereby savinglives andreducingcosts. Feature selection has become an essential component for developing Machinelearning models. It chooses the most relevant features from the available dataset,thereby shortening the training period, making the model easier to train, improving generalization and decreasing overfitting without necessarily compromising the system’s accuracy. Aim:The purpose of this work is to design and build an optimal model forthe prediction of heart diseases,especially at an early stage by considering certain features that are most relevant forthe prediction without compromising the system’s accuracy. Method: The Cleveland UCI dataset with 303 instances wereused in trainingthe model and the findings showthat selectKBest is an effective tool in improving the prediction of heart diseases. The performance metrics Accuracy, Sensitivity, Precision were measured.Results: the study found that when hybridizing k-Nearest Neighbor Bagging, Decision TreeBagging, Gradient Boosting generated the highest accuracy of 90%, 85% and 88% respectively.

Novel Enhanced-Grey Wolf Optimization hybrid machine learning technique for biomedical data computation

Today, a significant number of biomedical data is generated continuously from various biomedical equipment and experiments due to rapid technological improvements in medical sciences. Effective biomedical data analysis, such as extracting biological and diagnostically significant features, is a very challenging task. This paper proposes hybrid Machine Learning Classification Techniques based on ensemble technique with Enhanced-Grey Wolf Optimization (E-GWO) feature selection algorithm to analyze these complex biomedical data. We combined five biomedical heart disease data sets for the experimental work: Cleveland, Long-Beach-VA, Switzerland, Hungarians, and Statlog. New hybrid Machine Learning Classification Techniques classifiers like Naive Bayes Bagging Technique (NBBT), Random Forest Bagging Technique (RFBT), Decision Tree Bagging Technique (DTBT), K-Nearest Neighbors Bagging Technique (KNNBT), Neural Network Bagging Technique (NNBT), Gradient Boosting Boosting Technique (GBBT), and Adaptive Boosting Boosting Technique (ABBT) are developed with bagging and boosting methods. Accuracy, Recall, Precision, F1-score, Specificity, Error Rate, G-mean, False Negative Rate (FNR), False Positive Rate (FPR), and Negative Predictive Value (NPV) are used to evaluate hybrid techniques. The experimental result shows that the developed hybrid classifier RFBT achieves the highest accuracy of 99.26% with E-GWO feature selection. The proposed method has 11.90% improved the accuracy to the conventional model.

Predicting Gold and Silver Price Direction Using Tree-Based Classifiers

Gold is often used by investors as a hedge against inflation or adverse economic times. Consequently, it is important for investors to have accurate forecasts of gold prices. This paper uses several machine learning tree-based classifiers (bagging, stochastic gradient boosting, random forests) to predict the price direction of gold and silver exchange traded funds. Decision tree bagging, stochastic gradient boosting, and random forests predictions of gold and silver price direction are much more accurate than those obtained from logit models. For a 20-day forecast horizon, tree bagging, stochastic gradient boosting, and random forests produce accuracy rates of between 85% and 90% while logit models produce accuracy rates of between 55% and 60%. Stochastic gradient boosting accuracy is a few percentage points less than that of random forests for forecast horizons over 10 days. For those looking to forecast the direction of gold and silver prices, tree bagging and random forests offer an attractive combination of accuracy and ease of estimation. For each of gold and silver, a portfolio based on the random forests price direction forecasts outperformed a buy and hold portfolio.

Credal decision tree based novel ensemble models for spatial assessment of gully erosion and sustainable management

We introduce novel hybrid ensemble models in gully erosion susceptibility mapping (GESM) through a case study in the Bastam sedimentary plain of Northern Iran. Four new ensemble models including credal decision tree-bagging (CDT-BA), credal decision tree-dagging (CDT-DA), credal decision tree-rotation forest (CDT-RF), and credal decision tree-alternative decision tree (CDT-ADTree) are employed for mapping the gully erosion susceptibility (GES) with the help of 14 predictor factors and 293 gully locations. The relative significance of GECFs in modelling GES is assessed by random forest algorithm. Two cut-off-independent (area under success rate curve and area under predictor rate curve) and six cut-off-dependent metrics (accuracy, sensitivity, specificity, F-score, odd ratio and Cohen Kappa) were utilized based on both calibration as well as testing dataset. Drainage density, distance to road, rainfall and NDVI were found to be the most influencing predictor variables for GESM. The CDT-RF (AUSRC = 0.942, AUPRC = 0.945, accuracy = 0.869, specificity = 0.875, sensitivity = 0.864, RMSE = 0.488, F-score = 0.869 and Cohen’s Kappa = 0.305) was found to be the most robust model which showcased outstanding predictive accuracy in mapping GES. Our study shows that the GESM can be utilized for conserving soil resources and for controlling future gully erosion.

A Random Forests Approach to Predicting Clean Energy Stock Prices

Climate change, green consumers, energy security, fossil fuel divestment, and technological innovation are powerful forces shaping an increased interest towards investing in companies that specialize in clean energy. Well informed investors need reliable methods for predicting the stock prices of clean energy companies. While the existing literature on forecasting stock prices shows how difficult it is to predict stock prices, there is evidence that predicting stock price direction is more successful than predicting actual stock prices. This paper uses the machine learning method of random forests to predict the stock price direction of clean energy exchange traded funds. Some well-known technical indicators are used as features. Decision tree bagging and random forests predictions of stock price direction are more accurate than those obtained from logit models. For a 20-day forecast horizon, tree bagging and random forests methods produce accuracy rates of between 85% and 90% while logit models produce accuracy rates of between 55% and 60%. Tree bagging and random forests are easy to understand and estimate and are useful methods for forecasting the stock price direction of clean energy stocks.

IDENTIFY OF FRAUDULENT FINANCIAL OPERATIONS USING THE MACHINE LEARNING ALGORITHM

Current work is devoted to the problem of automatic detection of fraudulent financial transactions. The article describes the causes of fraudulent transactions their typical attributes, as well as the basic principle of detection. The concepts of fraudulent and honest transactions are defined. Examples of algorithms for determining suspicious financial transactions in antifraud systems are given. Modern approaches to monitoring and detecting cases of fraud in remote banking systems are considered. The positive and negative aspects of each approach are described. Particular attention is paid to the problem of optimal recognition of transaction classes in highly unbalanced data. Methods for solving the problem of unbalanced data are considered. The choice of means for evaluating the operation of the machine learning model is justified considering the specifics of data distribution. As a solution, we propose an approach based on the use of ensemble classifiers in conjunction with balanced sampling algorithms, the key feature of which is to create a balanced sample not for the entire classifier, but for each student in the ensemble separately. Based on data on fraud in the field of bank credit cards, a comparison is made and the best classifier is selected among such ensemble algorithms as random forest, adaptive boosting and bagging of decision trees. To create balanced subsets of evaluators of ensemble algorithms, the algorithm of random insufficient sampling is used. To search for the optimal parameters of the classifiers, the random search algorithm on the grid is used. The results of experimental comparison of the selected methods are presented. The advantages of the proposed approach are analyzed, and the boundaries of its applicability are discussed.

Record linkage for farm-level data analytics: Comparison of deterministic, stochastic and machine learning methods

The advent of big data in agriculture increased the necessity of extracting useful information from large data collections. This knowledge is critical in optimizing production systems, while also addressing prevailing issues such as sustainability. One of the first, yet crucial, data analytics steps comprises integration. Integrated data from different sources can provide enhanced insight, as they may retain complementary information on the same entity. In the absence of a “unique universal identifier” to link entities (e.g. farms) from different databases, it is necessary to rely on their recorded attributes (e.g. farm name, owner). We propose a fully automated framework to match farms, across different datasets (i.e. farm matching) in a big data context. To assess performance, we used information on Brazilian beef cattle farms from two large datasets: 44,566 farms that made purchases at an animal nutrition company, and 32,776 that processed cattle at a meat packing company. Geographical search space reduction was implemented as an alternative to reduce the number of comparisons evaluated. To compare attributes between farm pairs, we contrasted two edit-based approaches, the Levenshtein and Jaro-Winkler metrics. We also compare deterministic, stochastic, and machine learning (ML) approaches, for classification of farm pairs as match or non-match. These techniques have been used in other record linkage domains. The deterministic approach requires all attributes to match exactly. The probabilistic approaches tested were Epi-Weights (CR) and Fellegi-Slunter (FS). Unsupervised ML approaches were k-means and bagged clustering (BC). Supervised methods were recursive partitioning trees, bagging of decision trees, bootstrap based classification trees, stochastic boosting, support vector machines (SVM), single-layer neural networks and logistic regression. Labels were produced by specialist review for both a training set of 295,012 comparison pairs and a testing set of 32,780. All techniques were evaluated in terms of testing set quality (accuracy, precision, sensitivity, and specificity) and completeness (number of matches) as well as efficiency (run-time). ML approaches outperformed the deterministic matching, which was superior than probabilistic methods. Within ML approaches, supervised methods outperformed unsupervised (except for BC). The best string metric was the Levenshtein. The best classification method in terms of quality and completeness was SVM (accuracy = 99.9%, precision = 91.1%, sensitivity = 97.3%, specificity = 99.9%), followed by BC (accuracy = 99.9%, precision = 90.8%, sensitivity = 93.2%, specificity = 99.9%). Results indicate that both SVM and BC are suitable for farm matching in scenarios where training labels are available, or not, respectively.

Reliable diagnostics using wireless sensor networks

Monitoring activities in industry may require the use of wireless sensor networks, for instance due to difficult access or hostile environment. But it is well known that this type of networks has various limitations like the amount of disposable energy. Indeed, once a sensor node exhausts its resources, it will be dropped from the network, stopping so to forward information about maybe relevant features towards the sink. This will result in broken links and data loss which impacts the diagnostic accuracy at the sink level. It is therefore important to keep the network's monitoring service as long as possible by preserving the energy held by the nodes. As packet transfer consumes the highest amount of energy comparing to other activities in the network, various topologies are usually implemented in wireless sensor networks to increase the network lifetime. In this paper, we emphasize that it is more difficult to perform a good diagnostic when data are gathered by a wireless sensor network instead of a wired one, due to broken links and data loss on the one hand, and deployed network topologies on the other hand. Three strategies are considered to reduce packet transfers: (1) sensor nodes send directly their data to the sink, (2) nodes are divided by clusters, and the cluster heads send the average of their clusters directly to the sink, and (3) averaged data are sent from cluster heads to cluster heads in a hop-by-hop mode, leading to an avalanche of averages. Their impact on the diagnostic accuracy is then evaluated. We show that the use of random forests is relevant for diagnostics when data are aggregated through the network and when sensors stop to transmit their values when their batteries are emptied. This relevance is discussed qualitatively and evaluated numerically by comparing the random forests performance to state-of-the-art PHM approaches, namely: basic bagging of decision trees, support vector machine, multinomial naive Bayes, AdaBoost, and Gradient Boosting. Finally, a way to couple the two best methods, namely the random forests and the gradient boosting, is proposed by finding the best hyperparameters of the former by using the latter.

Undersampling and Bagging of Decision Trees in the Analysis of Cardiorespiratory Behavior for the Prediction of Extubation Readiness in Extremely Preterm Infants.

Extremely preterm infants often require endotracheal intubation and mechanical ventilation during the first days of life. Due to the detrimental effects of prolonged invasive mechanical ventilation (IMV), clinicians aim to extubate infants as soon as they deem them ready.Unfortunately, existing strategies for prediction of extubation readiness vary across clinicians and institutions, and lead to high reintubation rates. We present an approach using Random Forest classifiers for the analysis of cardiorespiratory variability to predict extubation readiness. We address the issue of data imbalance by employing random undersampling of examples from the majority class before training each Decision Tree in a bag. By incorporating clinical domain knowledge, we further demonstrate that our classifier could have identified 71% of infants who failed extubation, while maintaining a success detection rate of 78%.

Respiration Disorders Classification With Informative Features for m-Health Applications.

Respiratory disorder is a highly prevalent condition associated with many adverse health problems. As the current means of diagnosis are obtrusive and ill-suited for real-time m-health applications, we explore a convenient and low-cost automatic approach that uses wearable microelectromechanical system sensor technology. The proposed system introduces the use of motion sensors to detect the changes in the anterior-posterior diameter of the chest wall during breathing function as well as extracting the informative respiratory features to be used for breathing disorders classification. Extensive evaluations are provided on six well-known classifiers with novel feature extraction techniques to distinguish among eight different pathological breathing patterns. The effects of the number of sensors, sensor placement, as well as feature selection on the classification performance are discussed. The experimental results conducted with ten subjects show the best accuracy rates of 97.50% by support vector machine and 97.37% with decision tree bagging (DTB) with all features and after feature selection, correspondingly. Furthermore, a binary classification is proposed for distinguishing between healthy people and patients with breath problems. The different assessments of classification parameters are provided by measuring the accuracy, sensitivity, specificity, F1-score and Mathew correlation coefficient. The accuracy rates above 98% suggest superior performance of DTB in binary recognition supported by the suggested new features.

The Assessment of Machine Learning Model Performance for Predicting Alluvial Deposits Distribution

This paper discusses the development and evaluation of distribution models for predicting alluvial mineral potential mapping. A number of existing models includes Weight of Evidence, Knowledge-driven Fuzzy, Data-driven Fuzzy, Neural-Network, Bayesian Classifier and Geostatistical Kriging. We offer classification models developed in our laboratory, where point pattern analysis was used to identify presence or absence of a known secondary alluvial (cassiterite) deposits in the Nigerian Younger Granite Region (NYGR) and the model performance assessed. We focused on the training and testing data split using longitudinal spatial data splitting (strips and halves) to ensure predictive attribute's independence. The spatial data split runs counter to the traditional random sample data selection as a procedure for checking overfitting of models mainly due to spatial data autocorrelation. Specifically, we used classification algorithms such as; Naive Bayes, Support Vector Machine, K-Nearest Neighbour, Decision Tree Bagging and Discriminant Analysis algorithms for training and testing. We analysed the model's performance results using model predictive accuracy and ROC curve values in two different approaches that improve spatial data independence among predictive attributes to give a meaningful model performance.

Real-time detection of transient cardiac ischemic episodes from ECG signals

We propose a new algorithm to detect and classify transient cardiac ischemia episodes, designed with the goal of providing a real-time execution without penalizing the classifier accuracy much. The algorithm is based on a novel mixture of time-domain analysis and machine learning techniques, specifically bagging of decision trees, and it has been developed using a well-recognized and freely distributed database, namely the long-term ST database. The ST episode detection sensitivity/positive predictivity using the annotation protocol A for this database is 68.26%/74.91%. The sensitivity result increases until 93.97% for the most dangerous episodes in terms of duration and magnitude (annotated according to protocol C). The test of the algorithm over the freely distributed part of the European Society of Cardiology database has shown results of sensitivity and positive predictivity of 83.33% and 77.31%, respectively. Those results are close to the results obtained by related works that present approaches to detect ischemia episodes off-line, which is remarkable if we take into account that in our real-time approach, less information is available during the classification process.

Classifying Amharic webnews

We present work aimed at compiling an Amharic corpus from the Web and automatically categorizing the texts. Amharic is the second most spoken Semitic language in the World (after Arabic) and used for countrywide communication in Ethiopia. It is highly inflectional and quite dialectally diversified. We discuss the issues of compiling and annotating a corpus of Amharic news articles from the Web. This corpus was then used in three sets of text classification experiments. Working with a less-researched language highlights a number of practical issues that might otherwise receive less attention or go unnoticed. The purpose of the experiments has not primarily been to develop a cutting-edge text classification system for Amharic, but rather to put the spotlight on some of these issues. The first two sets of experiments investigated the use of Self-Organizing Maps (SOMs) for document classification. Testing on small datasets, we first looked at classifying unseen data into 10 predefined categories of news items, and then at clustering it around query content, when taking 16 queries as class labels. The second set of experiments investigated the effect of operations such as stemming and part-of-speech tagging on text classification performance. We compared three representations while constructing classification models based on bagging of decision trees for the 10 predefined news categories. The best accuracy was achieved using the full text as representation. A representation using only the nouns performed almost equally well, confirming the assumption that most of the information required for distinguishing between various categories actually is contained in the nouns, while stemming did not have much effect on the performance of the classifier.

Classifier hierarchy learning by means of genetic algorithms

Classifier combination falls in the so called data mining area. Its aim is to combine some paradigms from the supervised classification – sometimes with a previous non-supervised data division phase – in order to improve the individual accuracy of the component classifiers. Formation of classifier hierarchies is an alternative among the several methods of classifier combination. In this paper we present a novel method to find good hierarchies of classifiers for given databases. In this new proposal, a search is performed by means of genetic algorithms, returning the best individual according to the classification accuracy over the dataset, estimated through 10-fold cross-validation. Experiments have been carried out over 14 databases from the UCI repository, showing an improvement in the performance compared to the single classifiers. Moreover, similar or better results than other approaches, such as decision tree bagging and boosting, have been obtained.