Decision Tree Induction Research Articles

Introducing students to machine learning with decision trees using CODAP and Jupyter Notebooks

AbstractThis paper reports on progress in the development of a teaching module on machine learning with decision trees for secondary‐school students, in which students use survey data about media use to predict who plays online games frequently. This context is familiar to students and provides a link between school and everyday experience. In this module, they use CODAP's “Arbor” plug‐in to manually build decision trees and understand how to systematically build trees based on data. Further on, the students use a menu‐based environment in a Jupyter Notebook to apply an algorithm that automatically generates decision trees and to evaluate and optimize the performance of these. Students acquire technical and conceptual skills but also reflect on personal and social aspects of the uses of algorithms from machine learning.

Prediction of False-Positive Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Molecular Results in a High-Throughput Open-Platform System

Widespread high-throughput testing for identification of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection by RT-PCR has been a foundation in the response to the coronavirus disease 2019 (COVID-19) pandemic. Quality assurance metrics for these RT-PCR tests are still evolving as testing is widely implemented. As testing increases, it is important to understand performance characteristics and the errors associated with these tests. Herein, we investigate a high-throughput, laboratory-developed SARS-CoV-2 RT-PCR assay to determine whether modeling can generate quality control metrics that identify false-positive (FP) results due to contamination. This study reviewed repeated clinical samples focusing on positive samples that test negative on re-extraction and PCR, likely representing false positives. To identify and predict false-positive samples, we constructed machine learning–derived models based on the extraction method used. These models identified variables associated with false-positive results across all methods, with sensitivities for predicting FP results ranging between 67% and 100%. Application of the models to all results predicted a total FP rate of 0.08% across all samples, or 2.3% of positive results, similar to reports for other RT-PCR tests for RNA viruses. These models can predict quality control parameters, enabling laboratories to generate decision trees that reduce interpretation errors, allow for automated reflex testing of samples with a high FP probability, improve workflow efficiency, and increase diagnostic accuracy for patient care.

WITHDRAWN: EGIA: A new node splitting method for decision tree generation: Special application in software fault prediction

Software Fault Prediction (SFP) is a method of estimation of faulty modules of the software. It helps developers, testers as well as maintainers to develop good quality software modules with proper refactoring and less complexity. Many classification models have been developed by researchers to classify faulty modules in software like Decision Tree, SVM, ANN, and Random Forest, etc. Out of these techniques, the Decision Tree is one of the widely used approaches, which works on the greedy strategy for node splitting. Many alternates of node splitting have been attempted in the recent past to generate the decision tree for SFP such as Gini-Index, CHID, Gain Ratio, and Information Gain, etc. each of these has its pros and cons, but none of these individually perform well in terms of accuracy and/or efficiency. This paper proposes a new node splitting method based on combination of Gini-Index and Entropy for generating the decision tree. A new node splitting method called EGIA has been developed by integrating both node-splitting alternates. The proposed approach has been tested on 18 open source datasets of software implemented in different languages (C, C++, and Java). Results show that the proposed approach generates more accurate decision trees and can predict the software faulty modules better than the existing approaches. At the same time, the proposed method of node splitting is more dynamic than previous alternates because it can be modified according to the user's need and can be integrated with different optimization methods.

Predicting and interpreting oxide glass properties by machine learning using large datasets

With the advent of powerful computer simulation techniques, it is time to move from the widely used knowledge-guided empirical methods to approaches driven by data science, mainly machine learning algorithms. We investigated the predictive performance of three machine learning algorithms for six different glass properties. For such, we used an extensive dataset of about 150,000 oxide glasses, which was segmented into smaller datasets for each property investigated. Using the decision tree induction, k-nearest neighbors, and random forest algorithms, selected from a previous study of six algorithms, we induced predictive models for glass transition temperature, liquidus temperature, elastic modulus, thermal expansion coefficient, refractive index, and Abbe number. Moreover, each model was induced with default and tuned hyperparameter values. We demonstrate that, apart from the elastic modulus (which had the smallest training dataset), the induced predictive models for the other five properties yield a comparable uncertainty to the usual data spread. However, for glasses with extremely low or high values of these properties, the prediction uncertainty is significantly higher. Finally, as expected, glasses containing chemical elements that are poorly represented in the training set yielded higher prediction errors. The method developed here calls attention to the success and possible pitfalls of machine learning algorithms. The analysis of the SHAP values indicated the key elements that increase or decrease the value of the modeled properties. It also estimated the maximum possible increase or decrease. Insights gained by this analysis can help empirical compositional tuning and computer-aided inverse design of glass formulations.

Hamstring and gluteal activation during high-speed overground running: Impact of prior strain injury

ABSTRACT This study examined the spatial patterns of hamstring and gluteal muscle activation during high-speed overground running in limbs with and without aprior hamstring strain injury. Ten active males with arecent (<18 month) unilateral biceps femoris long head (BFLH) strain injury underwent functional magnetic resonance imaging before and immediately after arepeat-sprint running protocol. Transverse relaxation (T2) time, an index of muscle activation, of the BFLH and short head (BFSH), semitendinosus (ST), semimembranosus (SM), gluteus maximus (GMAX) and medius (GMED) was assessed pre-post exercise. No significant between-limb differences in running-induced mean T2 changes were observed (p = 0.949), however, decision tree induction revealed that previously injured limbs were characterised by highly variable intramuscular activation of the ST (SD5.3). T2 times increased more for GMAX than all other muscles (all p< 0.001, d= 0.5–2.5). Further, T2 changes were greater for ST than BFSH, SM, GMED, and BFLH (all p≤ 0.001, d= 0.5–2.9); and were greater for BFLH than BFSH, SM, and GMED (all p< 0.001, d= 1.2–1.6). Athletes display heterogenous patterns of posterior thigh activation when sprinting (GMAX>ST>BFLH>GMED>SM>BFSH) and may exhibit altered intramuscular hamstring activation after returning to sport from BFLH strain injury.

Fault Detection and Diagnosis in Air Handling Units with a Novel Integrated Decision Tree Algorithm

In air handling units (AHUs), wide attention has been attracted by data-driven fault detection and diagnosis techniques as the need for high-level expert knowledge of the concerned system is eliminated. In AHUs, decision tree induction is performed by means of classification and regression tree algorithm which is a data-driven diagnostic strategy based on decision tree. Expert knowledge as well as testing data may be used for validation of fault diagnosis reliability with easy interpretation and understanding ability offered by the decision tree. The diagnostic strategy established and its interpretability are increased by incorporating a regression model and steady-state detector with the model. ASHRAE, Oak Ridge National Lab (ORNL), National Renewable Energy Lab (NREL), Pacific Northwest National Lab (PNNL) and Lawrence Berkeley National Lab (LBNL) datasets are used for validation of the proposed strategy. High average F-measure and improved diagnostic performance may be achieved with this strategy. There is a compliance between the expert knowledge and certain diagnostic rules generated in the decision tree as seen from the expert knowledge implemented diagnostic decision tree interpretation. Based on the interpretation, it is evident that certain diagnostic rules are valid only under specific operating conditions and some of the generated diagnostic rules are not reliable. Data driven models are used for emphasizing the significance of interpretability of fault diagnostic models.

Counting the Number of Perfect Matchings, and Generalized Decision Trees

Counting the Number of Perfect Matchings, and Generalized Decision Trees

Machine-learning algorithm as a prognostic tool in non-obstructive acute-on-chronic kidney disease in the cat.

The aim of this study was to develop an algorithm capable of predicting short- and medium-term survival in cases of intrinsic acute-on-chronic kidney disease (ACKD) in cats. The medical record database was searched to identify cats hospitalised for acute clinical signs and azotaemia of at least 48 h duration and diagnosed to have underlying chronic kidney disease based on ultrasonographic renal abnormalities or previously documented azotaemia. Cases with postrenal azotaemia, exposure to nephrotoxicants, feline infectious peritonitis or neoplasia were excluded. Clinical variables were combined in a clinical severity score (CSS). Clinicopathological and ultrasonographic variables were also collected. The following variables were tested as inputs in a machine learning system: age, body weight (BW), CSS, identification of small kidneys or nephroliths by ultrasonography, serum creatinine at 48 h (Crea48), spontaneous feeding at 48 h (SpF48) and aetiology. Outputs were outcomes at 7, 30, 90 and 180 days. The machine-learning system was trained to develop decision tree algorithms capable of predicting outputs from inputs. Finally, the diagnostic performance of the algorithms was calculated. Crea48 was the best predictor of survival at 7 days (threshold 1043 µmol/l, sensitivity 0.96, specificity 0.53), 30 days (threshold 566 µmol/l, sensitivity 0.70, specificity 0.89) and 90 days (threshold 566 µmol/l, sensitivity 0.76, specificity 0.80), with fewer cats still alive when their Crea48 was above these thresholds. A short decision tree, including age and Crea48, predicted the 180-day outcome best. When Crea48 was excluded from the analysis, the generated decision trees included CSS, age, BW, SpF48 and identification of small kidneys with an overall diagnostic performance similar to that using Crea48. Crea48 helps predict short- and medium-term survival in cats with ACKD. Secondary variables that helped predict outcomes were age, CSS, BW, SpF48 and identification of small kidneys.

A branch & bound algorithm to determine optimal bivariate splits for oblique decision tree induction

Univariate decision tree induction methods for multiclass classification problems such as CART, C4.5 and ID3 continue to be very popular in the context of machine learning due to their major benefit of being easy to interpret. However, as these trees only consider a single attribute per node, they often get quite large which lowers their explanatory value. Oblique decision tree building algorithms, which divide the feature space by multidimensional hyperplanes, often produce much smaller trees but the individual splits are hard to interpret. Moreover, the effort of finding optimal oblique splits is very high such that heuristics have to be applied to determine local optimal solutions. In this work, we introduce an effective branch and bound procedure to determine global optimal bivariate oblique splits for concave impurity measures. Decision trees based on these bivariate oblique splits remain fairly interpretable due to the restriction to two attributes per split. The resulting trees are significantly smaller and more accurate than their univariate counterparts due to their ability of adapting better to the underlying data and capturing interactions of attribute pairs. Moreover, our evaluation shows that our algorithm even outperforms algorithms based on heuristically obtained multivariate oblique splits despite the fact that we are focusing on two attributes only.

Establishing a Multiple-Criteria Decision-Making Model for Stock Investment Decisions Using Data Mining Techniques

This study attempts to integrate the decision tree algorithm with the Apriori algorithm to explore the relationship among financial ratio, corporate governance, and stock returns to establish a stock investment decision model. The sports and leisure related industries are employed as the research target. The data are collected and processed for generating decision tree and association rules. Based on the analysis outcome, an investment decision model is constructed for investors expecting to decrease their investment risks and further increase their profits. This stock investment decision model is one type of multiple-criteria decision-making model. This study makes three critical contributions to investors. (1) It proposes a systematical model of exploring related data through the decision tree algorithm and the Apriori algorithm to reveal the implicit investment knowledge. (2) An effective investment decision model is established and expected to provide a reference basis during stock-picking decisions. (3) The investment decision model is enhanced with implicit rules found among variables using association rules.

Root System and Its Relations with Soil Physical and Chemical Attributes in Orange Culture

Citrus companies have sought and developed alternative systems of tillage or implanting orchards so as not to significantly alter the physical and chemical attributes of the soil and, consequently, the root development of plants. Therefore, the aim of this study was to identify the physical and chemical attributes of the soil that most influence the root volume of the orange crop in different tillage systems. The experiment was carried out in the region of Avaré, state of São Paulo, Brazil, in Utissol and Oxisol. For the planting of the orange crop, the following tillages were made: minimum tillage, subsoiler tillage and soil tillage using a triple tillage implement. The physical and chemical attributes evaluated were bulk density, macroporosity, microporosity, total porosity, soil moisture, soil mean weight-diameter, soil resistance to penetration, sum of bases, cation exchange capacity, base saturation, pH, exchangeable cations, potential acidity, available phosphorus, analysis of micronutrients such as copper, iron, manganese and zinc, and organic carbon content. The root system was evaluated using SIARCS® software. For data classification, data mining techniques were used such as attribute selection and decision tree induction. Regardless of the soil type, the use of the triple operation implement provided greater root volume for orange plants. For the Utisol area, the pH value of 4.2 was the main attribute that provided a high root volume. For the Oxisol, the presence of copper, in levels that did not generate toxicity for the plants, provided a high volume of root for the crop.

Optimization of decision trees using modified African buffalo algorithm

Decision tree induction is a simple, however powerful learning and classification tool to discover knowledge from the database. The volume of data in databases is growing to quite large sizes, both in the number of attributes and instances. Some important limitations of decision trees are instability, local decisions, and overfitting for this extensive data. The simple, effective and non-convergence nature of the African Buffalo Optimization (ABO) algorithm makes it suitable to solve complex optimization problems. In this paper, we propose the African Buffalo Optimized Decision Tree (ABODT) algorithm to create globally optimized decision trees using the intelligent and collective behaviour of African Buffalos. The modified African Buffalo optimization algorithm is used to create efficient and optimal decision trees. To evaluate the efficiency of the proposed African Buffalo Optimized Decision Tree algorithm, experiments are performed on 15 standard UCI learning repository datasets that are of various sizes and domains. Results show that the African Buffalo Optimized Decision Tree algorithm globally optimizes decision trees, increases accuracy and reduces the size of a decision tree. These optimized trees are stable and efficient than conventional decision trees.

SALAD-BAAR: A numerical risk score for hospital admission or emergency department presentation in ambulatory patients with cardiovascular disease.

BackgroundWhile many interventions to reduce hospital admissions and emergency department (ED) visits for patients with cardiovascular disease have been developed, identifying ambulatory cardiac patients at high risk for admission can be challenging.HypothesisA computational model based on readily accessible clinical data can identify patients at risk for admission.MethodsElectronic health record (EHR) data from a tertiary referral center were used to generate decision tree and logistic regression models. International Classification of Disease (ICD) codes, labs, admissions, medications, vital signs, and socioenvironmental variables were used to model risk for ED presentation or hospital admission within 90 days following a cardiology clinic visit. Model training and testing were performed with a 70:30 data split. The final model was then prospectively validated.ResultsA total of 9326 patients and 46 465 clinic visits were analyzed. A decision tree model using 75 patient characteristics achieved an area under the curve (AUC) of 0.75 and a logistic regression model achieved an AUC of 0.73. A simplified 9‐feature model based on logistic regression odds ratios achieved an AUC of 0.72. A further simplified numerical score assigning 1 or 2 points to each variable achieved an AUC of 0.66, specificity of 0.75, and sensitivity of 0.58. Prospectively, this final model maintained its predictive performance (AUC 0.63–0.60).ConclusionNine patient characteristics from routine EHR data can be used to inform a highly specific model for hospital admission or ED presentation in cardiac patients. This model can be simplified to a risk score that is easily calculated and retains predictive performance.

A Practical Tutorial for Decision Tree Induction

Experts from different domains have resorted to machine learning techniques to produce explainable models that support decision-making. Among existing techniques, decision trees have been useful in many application domains for classification. Decision trees can make decisions in a language that is closer to that of the experts. Many researchers have attempted to create better decision tree models by improving the components of the induction algorithm. One of the main components that have been studied and improved is the evaluation measure for candidate splits.In this article, we introduce a tutorial that explains decision tree induction. Then, we present an experimental framework to assess the performance of 21 evaluation measures that produce different C4.5 variants considering 110 databases, two performance measures, and 10× 10-fold cross-validation. Furthermore, we compare and rank the evaluation measures by using a Bayesian statistical analysis. From our experimental results, we present the first two performance rankings in the literature of C4.5 variants. Moreover, we organize the evaluation measures into two groups according to their performance. Finally, we introduce meta-models that automatically determine the group of evaluation measures to produce a C4.5 variant for a new database and some further opportunities for decision tree models.

Improving measurements of similarity judgments with machine-learning algorithms

Intertemporal choices involve assessing options with different reward amounts available at different time delays. The similarity approach to intertemporal choice focuses on judging how similar amounts and delays are. Yet we do not fully understand the cognitive process of how these judgments are made. Here, we use machine-learning algorithms to predict similarity judgments to (1) investigate which algorithms best predict these judgments, (2) assess which predictors are most useful in predicting participants’ judgments, and (3) determine the minimum number of judgments required to accurately predict future judgments. We applied eight algorithms to similarity judgments for reward amount and time delay made by participants in two data sets. We found that neural network, random forest, and support vector machine algorithms generated the highest out-of-sample accuracy. Though neural networks and support vector machines offer little clarity in terms of a possible process for making similarity judgments, random forest algorithms generate decision trees that can mimic the cognitive computations of human judgment making. We also found that the numerical difference between amount values or delay values was the most important predictor of these judgments, replicating previous work. Finally, the best performing algorithms such as random forest can make highly accurate predictions of judgments with relatively small sample sizes (~ 15), which will help minimize the numbers of judgments required to extrapolate to new value pairs. In summary, machine-learning algorithms provide both theoretical improvements to our understanding of the cognitive computations involved in similarity judgments and intertemporal choices as well as practical improvements in designing better ways of collecting data.

Knowledge discovery in open data for epidemic disease prediction

ObjectiveThe research reveals the determinants associated with the epidemic diseases (dengue, influenza, and enterovirus) in Taiwan. It demonstrates the value of open data in prediction model development to support policymaking in the domain of public health care. MethodA knowledge discovery technique was employed to extract determinants from open data on epidemic diseases. The open dataset collected and integrated from Taiwan's Center for Disease Control, the Center Weather Bureau, and Google Trends includes 70,915 dengue, 34,062 enterovirus, and 52,908 influenza cases. A prediction model using the classification-oriented extraction mechanism was applied to open epidemic data, climate data, and Google Trends data. Prediction models that either included or did not include Google Trends data were compared. Prediction accuracy and simplicity of the decision rules are presented. ResultsPrediction accuracy and simplicity of three diseases is acceptable when Google Trends is excluded but is slightly different when Google Trends is considered. Location (county) holds the main predictor of the three epidemic diseases. Time (month) presents the second-highest determinant for dengue, and age shows remarkable determinant for enterovirus and influenza. Mean temperature exhibits the highest entropy for dengue, time for enterovirus, and humidity for influenza. ConclusionsThe number of confirmed cases for all three epidemic diseases cannot be predicted by a single variable. Knowledge extraction using the classification-oriented technique can be successfully applied in prediction model development. Google Trends data reveal a remarkable but inconsistent role in predicting three epidemic diseases with respect to prediction accuracy and simplicity of the generated decision tree.

Multi-GPU approach to global induction of classification trees for large-scale data mining

This paper concerns the evolutionary induction of decision trees (DT) for large-scale data. Such a global approach is one of the alternatives to the top-down inducers. It searches for the tree structure and tests simultaneously and thus gives improvements in the prediction and size of resulting classifiers in many situations. However, it is the population-based and iterative approach that can be too computationally demanding to apply for big data mining directly. The paper demonstrates that this barrier can be overcome by smart distributed/parallel processing. Moreover, we ask the question whether the global approach can truly compete with the greedy systems for large-scale data. For this purpose, we propose a novel multi-GPU approach. It incorporates the knowledge of global DT induction and evolutionary algorithm parallelization together with efficient utilization of memory and computing GPU’s resources. The searches for the tree structure and tests are performed simultaneously on a CPU, while the fitness calculations are delegated to GPUs. Data-parallel decomposition strategy and CUDA framework are applied. Experimental validation is performed on both artificial and real-life datasets. In both cases, the obtained acceleration is very satisfactory. The solution is able to process even billions of instances in a few hours on a single workstation equipped with 4 GPUs. The impact of data characteristics (size and dimension) on convergence and speedup of the evolutionary search is also shown. When the number of GPUs grows, nearly linear scalability is observed what suggests that data size boundaries for evolutionary DT mining are fading.

AdaDT: An adaptive decision tree for addressing local class imbalance based on multiple split criteria

As it is well known, decision tree is a kind of data-driven classification model, and its primary core is the split criterion. Although a great deal of split criteria have been proposed so far, almost all of them focus on the global class distribution of the training data. However, they ignored the local class imbalance problem that commonly appears during the decision tree induction over balanced or roughly balanced binary class data sets. In the present study, this problem is investigated in detail and an adaptive approach based on multiple existing split criteria is proposed. In the proposed scheme, the local class imbalanced ratio is considered as the weight factor to weigh the importance between these split criteria so as to determine the optimal splitting point at each internal node. In order to evaluate the effectiveness of the proposed method, it is applied on twenty roughly balanced real-world binary class data sets. Experimental results show that the proposed method not only outperforms all other methods, but also improves the prediction accuracy of each class.

Optimisation and evaluation of the random forest model in the efficacy prediction of chemoradiotherapy for advanced cervical cancer based on radiomics signature from high-resolution T2 weighted\xa0images

PurposeOur objective was to establish a random forest model and to evaluate its predictive capability of the treatment effect of neoadjuvant chemotherapy–radiation therapy.MethodsThis retrospective study included 82 patients with locally advanced cervical cancer who underwent scanning from March 2013 to May 2018. The random forest model was established and optimised based on the open source toolkit scikit-learn. Byoptimising of the number of decision trees in the random forest, the criteria for selecting the final partition index and the minimum number of samples partitioned by each node, the performance of random forest in the prediction of the treatment effect of neoadjuvant chemotherapy–radiation therapy on advanced cervical cancer (> IIb) was evaluated.ResultsThe number of decision trees in the random forests influenced the model performance. When the number of decision trees was set to 10, 25, 40, 55, 70, 85 and 100, the performance of random forest model exhibited an increasing trend first and then a decreasing one. The criteria for the selection of final partition index showed significant effects on the generation of decision trees. The Gini index demonstrated a better effect compared with information gain index. The area under the receiver operating curve for Gini index attained a value of 0.917.ConclusionThe random forest model showed potential in predicting the treatment effect of neoadjuvant chemotherapy–radiation therapy based on high-resolution T2WIs for advanced cervical cancer (> IIb).

Privacy preserving techniques for decision trees

As a representative classification model, decision tree has been extensively applied in data mining. It generates a series of if-then rules based on the homogeneity of class distribution. In a society where data spreads everywhere for knowledge discovery, the privacy of the data respondents is likely to be leaked and abused. Based on this concern, we propose an overview of the rapidly evolving research results focusing on privacy preserving decision tree induction. The research results are summarised according to the characteristics of related privacy preservation techniques, which include data perturbation, cryptography, and data anonymisation. In addition, we demonstrate the comparison between the merits and demerits of these methods considering the specific property of decision tree induction. At last, we conclude the future trend of privacy preserving techniques.