Decision Tree Induction Method Research Articles

An Experimental Comparison of Self-Adaptive Differential Evolution Algorithms to Induce Oblique Decision Trees

This study addresses the challenge of generating accurate and compact oblique decision trees using self-adaptive differential evolution algorithms. Although traditional decision tree induction methods create explainable models, they often fail to achieve optimal classification accuracy. To overcome these limitations, other strategies, such as those based on evolutionary computation, have been proposed in the literature. In particular, we evaluate the use of self-adaptive differential evolution variants to evolve a population of oblique decision trees encoded as real-valued vectors. Our proposal includes (1) an alternative initialization strategy that reduces redundant nodes and (2) a fitness function that penalizes excessive leaf nodes, promoting smaller and more accurate decision trees. We perform a comparative performance analysis of these differential evolution variants, showing that while they exhibit similar statistical behavior, the Single-Objective real-parameter optimization (jSO) method produces the most accurate oblique decision trees and is second best in compactness. The findings highlight the potential of self-adaptive differential evolution algorithms to improve the effectiveness of oblique decision trees in machine learning applications.

Bsnsing: A Decision Tree Induction Method Based on Recursive Optimal Boolean Rule Composition

This paper proposes a new mixed-integer programming (MIP) formulation to optimize split rule selection in the decision tree induction process and develops an efficient search algorithm that is able to solve practical instances of the MIP model faster than commercial solvers. The formulation is novel for it directly maximizes the Gini reduction, an effective split selection criterion that has never been modeled in a mathematical program for its nonconvexity. The proposed approach differs from other optimal classification tree models in that it does not attempt to optimize the whole tree; therefore, the flexibility of the recursive partitioning scheme is retained, and the optimization model is more amenable. The approach is implemented in an open-source R package named bsnsing. Benchmarking experiments on 75 open data sets suggest that bsnsing trees are the most capable of discriminating new cases compared with trees trained by other decision tree codes including the rpart, C50, party, and tree packages in R. Compared with other optimal decision tree packages, including DL8.5, OSDT, GOSDT, and indirectly more, bsnsing stands out in its training speed, ease of use, and broader applicability without losing in prediction accuracy. History: Accepted by RamRamesh, Area Editor for Data Science & Machine Learning. Funding: This work was supported by the National Science Foundation Division of Civil, MechanicalandManufacturing Innovation [Grant 1944068]. Supplemental Material: Data are available at https://doi.org/10.1287/ijoc.2022.1225 .

A branch & bound algorithm to determine optimal bivariate splits for oblique decision tree induction

Univariate decision tree induction methods for multiclass classification problems such as CART, C4.5 and ID3 continue to be very popular in the context of machine learning due to their major benefit of being easy to interpret. However, as these trees only consider a single attribute per node, they often get quite large which lowers their explanatory value. Oblique decision tree building algorithms, which divide the feature space by multidimensional hyperplanes, often produce much smaller trees but the individual splits are hard to interpret. Moreover, the effort of finding optimal oblique splits is very high such that heuristics have to be applied to determine local optimal solutions. In this work, we introduce an effective branch and bound procedure to determine global optimal bivariate oblique splits for concave impurity measures. Decision trees based on these bivariate oblique splits remain fairly interpretable due to the restriction to two attributes per split. The resulting trees are significantly smaller and more accurate than their univariate counterparts due to their ability of adapting better to the underlying data and capturing interactions of attribute pairs. Moreover, our evaluation shows that our algorithm even outperforms algorithms based on heuristically obtained multivariate oblique splits despite the fact that we are focusing on two attributes only.

Soil carbon stock in archaeological black earth under different land use systems in the Brazilian Amazon

AbstractIn the Amazon, there are soils associated with continued human occupation known as “archeological black earth” (ABE). Due to its physical and chemical properties, ABE is more productive than other typical soils in the same region. Therefore, its carbon (C) sequestration mechanism has been a major topic of discussion by the scientific community, aiming to replicate similar characteristics in other soils. Thus, the objective of this study was to develop a predictive model using feature selection and decision tree induction methods for predicting soil C stock in ABE under different land use scenarios. The experiment was carried out in agricultural (coffee, cacao, and beans), pasture, and forest areas. Four feature selection approaches were used to identify the most relevant variables for the proposed model: (i) correlation‐based feature selection, (ii) the χ2 test, (iii) the Wrapper method, and (iv) no feature selection. The decision tree induction technique available in the Weka software was selected for data classification. Soils under cacao and coffee cultivation tend to accumulate more C when compared with soils located at bean crops, pasture, or forest land use systems. Land use and sand content were among the most important variables for the prediction of soil C stock in ABE. Furthermore, the use of a decision tree was effective at predicting soil C stocks for these soils because it enables the creation of models with high accuracy rates of 83, 74, and 81% (using seven, seven, and four rules at depths of 0.00–0.05, 0.05–0.10, and 0.10–0.20 m, respectively).

Rule-Based Classification using Multi Soft Set Theory

One of the important data mining functionality is classification. Presently, different methods exist for implementing classification. Rule-based classification using decision tree induction method is a conventional and simple method for identifying an unknown class of a given object. This method has a set of demerits and to remove these demerits, we depend on a soft computing tool which is known as soft set theory. One branch of soft set theory is called - multi soft theory- and it has a wide range of applications in the area of classification. We made a certain alteration in the rule-based classification using decision tree induction method by applying multi soft set theory. These changes will simplify the difficulties of the rule-based classification using decision tree induction method. The first two sections of this research work discuss introduction and preliminaries. In the remaining sections, the authors describe the multi soft set theory and its applications in rule base classification. Lastly, the paper finishes with a new algorithm, which the research scholars implemented as software using python programming. The suggested work experts can use in data mining industry. It has massive use in the fields of business, agriculture, health, education and many more.

Application of the decision tree method in forensic-medical practice in the analysis of 'doctors proceedings'

The possibilities of Data Mining artificial intelligence technology, in particular the decision tree induction method, are presented to answer special questions during the forensic examination of criminal proceedings instituted against doctors in cases of inadequate medical care. Based on the results, justified by an automated mathematical program for intelligent database processing, specific forensic expert features of therapeutic and tactical defects in the provision of medical care are established.

Similarity-based decision tree induction method and its application to cancer recognition on tomographic images

The paper proposes a pattern recognition method using a modification of the class of logical decision functions presented in the form of decision tree. Instead of standard statements corresponding to the tree nodes, in which a variable is tested for a certain set of its values, a more general type of statements is used regarding the similarity of the point in question to different subsets of the observations. At the same time, to determine the degree of similarity, various metrics and subspaces of features can be used. This type of decision tree allows one to obtain more complex decision boundaries, which at the same time have a clear logical interpretation for the user. Several tree induction strategies are considered based on data transformation using support points selected with Relief, SVM, and k -means procedures. The method is experimentally investigated on the problem of tomographic images analysis, as well as on several synthetic datasets. Experiments have shown that the proposed method gives more accurate predictions than CART, SVM, kNN classifiers and deep convolutional neural network (AlexNet).

A Hybrid Data Mining Approach for Generalizing Characteristics of Emergency Department Visits Causing Overcrowding

Hospital emergency department (ED) crowding has led to an increase in patient waiting times; solving this problem requires a better understanding of the patient behavior. In this work, we adopt decision tree analysis which facilitate the interpretation and understanding of ED visits at Mackay Memorial Hospital, a representative ED in our country. Accordingly, a hybrid data mining approach is proposed to predict patients’ LOS and explain their associated characteristics under various LOSs, especially for frequent and non-urgent groups with shorter stays in the ED. With two datasets from the first half-years of 2009 and 2010 containing 40,849 and 43,708 records respectively, we verify the stability and robustness of the proposed approach. We confirm the qualified rules based on patient characteristics and treatment information extracted by the decision tree induction method for the patient population that primarily causes ED overcrowding—patients with non-urgent conditions and short ED stays—in terms of accuracy, medical clinical value and relatedness. We identify that patients with short LOSs demonstrated similar characteristics in visiting ED. We also identify that attributes such as treatment frequencies of laboratory testing, age, and mode of arrival are good indicators for predicting patients’ LOSs. The results clarify ED crowding in Taiwan and can guide investigations of ED overcrowding from the perspective of generalizing characteristics of visits. The results serve as a reference model for related ED research in a similar context for clinical decision support.Keywords: Clinical Decision Support; ED Overcrowding; Hybrid Data Mining; Length of Stay; Patient Characteristics

Construction of Near-Optimal Axis-Parallel Decision Trees Using a Differential-Evolution-Based Approach

In this paper, a differential-evolution-based approach implementing a global search strategy to find a near-optimal axis-parallel decision tree is introduced. In this paper, the internal nodes of a decision tree are encoded in a real-valued chromosome, and a population of them evolves using the training accuracy of each one as its fitness value. The height of a complete binary decision tree whose number of internal nodes is not less than the number of attributes in the training set is used to compute the chromosome size, and a procedure to map a feasible axis-parallel decision tree from one chromosome is applied, which uses both the smallest-position-value rule and the training instances. The best decision tree in the final population is refined replacing some leaf nodes with sub-trees to improve its accuracy. The differential evolution algorithm has been successfully applied in conjunction with several supervised learning methods to solve numerous classification problems, due to it exhibiting a good tradeoff between its exploitation and exploration skills, and to the best of our knowledge, it has not been utilized to build axis-parallel decision trees. To obtain reliable estimates of the predictive performance of this approach and to compare its results with those achieved by other methods, a repeated stratified ten-fold cross-validation procedure is applied in the experimental study. A statistical analysis of these results suggests that our approach is better as a decision tree induction method as compared with other supervised learning methods. Also our results are comparable to those obtained with random forest and one multilayer-perceptron-based classifier.

Large-scale data analysis on aviation accident database using different data mining techniques

ABSTRACTData mining is an iterative process in which progress is defined by discovery through either automatic or manual methods. A data cleaning procedure is proposed to improve the quality of classification tasks in the knowledge discovery process by taking into account both redundant and conflicting data. The redundancy check is performed on the original dataset and the resultant dataset is preserved. This resultant dataset is then checked for conflicting data and, if any are found, they are corrected and updated on the original aircraft dataset. This updated dataset is then classified using a variety of classifiers such as Bayes, functions, lazy, MISC, rules and decision trees. The performance of the updated datasets on these classifiers is examine, and the result shows a significant improvement in the classification accuracy after redundancy and conflicts are removed. The conflicts after correction are updated in the original dataset, and when the performance of the classifier is evaluated, great improvement is observed. This paper aims to address how data mining techniques can be used to understand complex system accidents in the aviation domain. Decision trees are considered to be the one of the most powerful and popular approaches in knowledge discovery and data mining. The objective is to develop a classification model for aviation risk investigation and reduction using a decision tree induction method that enhances the ability to form decision trees and thereby proves that the classification accuracy of decision trees is greater. Different feature selectors are used in this study in order to reduce the number of initial attributes.

Manifold Learning Co-Location Decision Tree for Remotely Sensed Imagery Classification

Because traditional decision tree (DT) induction methods cannot efficiently take advantage of geospatial knowledge in the classification of remotely sensed imagery, several researchers have presented a co-location decision tree (CL-DT) method that combines the co-location technique with the traditional DT method. However, the CL-DT method only considers the Euclidean distance of neighborhood events, which cannot truly reflect the co-location relationship between instances for which there is a nonlinear distribution in a high-dimensional space. For this reason, this paper develops the theory and method for a maximum variance unfolding (MVU)-based CL-DT method (known as MVU-based CL-DT), which includes unfolding input data, unfolded distance calculations, MVU-based co-location rule generation, and MVU-based CL-DT generation. The proposed method has been validated by classifying remotely sensed imagery and is compared with four other types of methods, i.e., CL-DT, classification and regression tree (CART), random forests (RFs), and stacked auto-encoders (SAE), whose classification results are taken as “true values.” The experimental results demonstrate that: (1) the relative classification accuracies of the proposed method in three test areas are higher than CL-DT and CART, and are at the same level compared to RFs; and (2) the total number of nodes, the number of leaf nodes, and the number of levels are significantly decreased by the proposed method. The time taken for the data processing, decision tree generation, drawing of the tree, and generation of the rules are also shortened by the proposed method compared to CL-DT, CART, and RFs.

Comparing four methods for decision-tree induction: A case study on the invasive Iberian gudgeon (Gobio lozanoi; Doadrio and Madeira, 2004)

The invasion of freshwater ecosystems is a particularly alarming phenomenon in the Iberian Peninsula. Habitat suitability modelling is a proficient approach to extract knowledge about species ecology and to guide adequate management actions. Decision-trees are an interpretable modelling technique widely used in ecology, able to handle strongly nonlinear relationships with high order interactions and diverse variable types. Decision-trees recursively split the input space into two parts maximising child node homogeneity. This recursive partitioning is typically performed with axis-parallel splits in a top-down fashion. However, recent developments of the R packages oblique.tree, which allows the development of oblique split-based decision-trees, and evtree, which performs globally optimal searches with evolutionary algorithms to do so, seem to outperform the standard axis-parallel top-down algorithms; CART and C5.0. To evaluate their possible use in ecology, the two new partitioning algorithms were compared with the two well-known, standard axis-parallel algorithms. The entire process was performed in R by simultaneously tuning the decision-tree parameters and the variables subset with a genetic algorithm and modelling the presence–absence of the Iberian gudgeon (Gobio lozanoi; Doadrio and Madeira, 2004), an invasive fish species that has spread across the Iberian Peninsula. The accuracy and complexity of the trees, the modelled patterns of mesohabitat selection and the variables importance were compared. None of the new R packages, namely oblique.tree and evtree, outperformed the C5.0 algorithm. They rendered almost the same decision-trees as the CART algorithm, although they were completely interpretable – they performed from four to eight partitions – in comparison with C5.0, which resulted in a more complex structure with 17 partitions. Oblique.tree proved to be affected by prevalence and it does not include the possibility of weighting the observations, which potentially discourage its actual use. Although the use of evtree did not suggest a major improvement compared with the remaining packages, it allowed the development of regression trees which may be informative for additional modelling tasks such as abundance estimation. Looking at the resulting decision-trees, the optimal habitats for the Iberian gudgeon were large pools in lowland river segments with depositional areas and aquatic vegetation present, which typically appeared in the form of scattered macrophytes clumps. Furthermore, Iberian gudgeon seems to avoid habitats characterised by scouring phenomena and limited vegetated cover availability. Accordingly, we can assume that river regulation and artificial impoundment would have favoured the spread of the Iberian gudgeon across the entire peninsula.

VR-BFDT: A variance reduction based binary fuzzy decision tree induction method for protein function prediction

In protein function prediction (PFP) problem, the goal is to predict function of numerous well-sequenced known proteins whose function is not still known precisely. PFP is one of the special and complex problems in machine learning domain in which a protein (regarded as instance) may have more than one function simultaneously. Furthermore, the functions (regarded as classes) are dependent and also are organized in a hierarchical structure in the form of a tree or directed acyclic graph. One of the common learning methods proposed for solving this problem is decision trees in which, by partitioning data into sharp boundaries sets, small changes in the attribute values of a new instance may cause incorrect change in predicted label of the instance and finally misclassification. In this paper, a Variance Reduction based Binary Fuzzy Decision Tree (VR-BFDT) algorithm is proposed to predict functions of the proteins. This algorithm just fuzzifies the decision boundaries instead of converting the numeric attributes into fuzzy linguistic terms. It has the ability of assigning multiple functions to each protein simultaneously and preserves the hierarchy consistency between functional classes. It uses the label variance reduction as splitting criterion to select the best “attribute-value” at each node of the decision tree. The experimental results show that the overall performance of the proposed algorithm is promising.

Comparing Data Mining Techniques in HIV Testing Prediction

Introduction: The present work compared the prediction power of the different data mining techniques used to develop the HIV testing prediction model. Four popular data mining algorithms (Decision tree, Naive Bayes, Neural network, logistic regression) were used to build the model that predicts whether an individual was being tested for HIV among adults in Ethiopia using EDHS 2011. The final experimentation results indicated that the decision tree (random tree algorithm) performed the best with accuracy of 96%, the decision tree induction method (J48) came out to be the second best with a classification accuracy of 79%, followed by neural network (78%). Logistic regression has also achieved the least classification accuracy of 74%. Objectives: The objective of this study is to compare the prediction power of the different data mining techniques used to develop the HIV testing prediction model. Methods: Cross-Industry Standard Process for Data Mining (CRISP-DM) was used to predict the model for HIV testing and explore association rules between HIV testing and the selected attributes. Data preprocessing was performed and missing values for the categorical variable were replaced by the modal value of the variable. Different data mining techniques were used to build the predictive model. Results: The target dataset contained 30,625 study participants. Out of which 16,515 (54%) participants were women while the rest 14,110 (46%) were men. The age of the participants in the dataset ranged from 15 to 59 years old with modal age of 15 - 19 years old. Among the study participants, 17,719 (58%) have never been tested for HIV while the rest 12,906 (42%) had been tested. Residence, educational level, wealth index, HIV related stigma, knowledge related to HIV, region, age group, risky sexual behaviour attributes, knowledge about where to test for HIV and knowledge on family planning through mass media were found to be predictors for HIV testing. Conclusion and Recommendation: The results obtained from this research reveal that data mining is crucial in extracting relevant information for the effective utilization of HIV testing services which has clinical, community and public health importance at all levels. It is vital to apply different data mining techniques for the same settings and compare the model performances (based on accuracy, sensitivity, and specificity) with each other. Furthermore, this study would also invite interested researchers to explore more on the application of data mining techniques in healthcare industry or else in related and similar settings for the future.

Interval-valued fuzzy decision trees with optimal neighbourhood perimeter

This research proposes a new model for constructing decision trees using interval-valued fuzzy membership values. Most existing fuzzy decision trees do not consider the uncertainty associated with their membership values, however, precise values of fuzzy membership values are not always possible. In this paper, we represent fuzzy membership values as intervals to model uncertainty and employ the look-ahead based fuzzy decision tree induction method to construct decision trees. We also investigate the significance of different neighbourhood values and define a new parameter insensitive to specific data sets using fuzzy sets. Some examples are provided to demonstrate the effectiveness of the approach.

A prospective field study for sensor-based identification of fall risk in older people with dementia

Objective: Aim of this study was to make a fall prognosis in a cohort of older people with dementia in short-term (2 month), mid-term (4 month) and long-term (8 month) intervals using accelerometry during the subjects’ everyday life.Methods: The study was designed as a longitudinal cohort study. The subjects were recruited from a nursing home and geriatric assessment tests were conducted at baseline. Each subject underwent four visits and was measured at each visit for one week. Gait episodes were detected and gait parameters were extracted from these episodes. These gait parameters were combined with the falls occurred during the study. A decision tree induction method was used to analyze the data.Results: Forty subjects participated in the study, whereby 12 drop-outs were registered. The geriatric assessment tests were unable to distinguish between the groups (AUC < 0.6). The evaluation of the models induced with the decision tree classification showed a rate of correctly classified gait episodes of 88.4% for short-term, 74.8% for mid-term, and 88.5 % for long-term monitoring.Discussion and conclusions: We concluded that it is possible to classify gait episodes of fallers and non-fallers in people with dementia during everyday life using accelerometry.

Fuzzy Decision Tree

Decision trees have been widely used in machine learning. However, due to some reasons, data collecting in real world contains a fuzzy and uncertain form. The decision tree should be able to handle such fuzzy data. This paper presents a method to construct fuzzy decision tree. It proposes a fuzzy decision tree induction method in iris flower data set, obtaining the entropy from the distance between an average value and a particular value. It also presents an experiment result that shows the accuracy compared to former ID3.

A Method for Supporting the Domain Expert by the Interpretation of Different Decision Trees Learnt from the Same Domain

Data mining methods are widely used across many disciplines to identify patterns, rules, or associations among huge volumes of data. Data mining methods with explanation capability such as decision tree induction are preferred in many domains. The aim of this paper is to discuss how to deal with the result of decision tree induction methods. This paper has been prompted by the fact that domain experts are able to use the tools for decision tree induction but have great difficulties in interpreting the results. When the domain expert has learnt two decision trees that are from the same domain but based on different data sets as a result of further data collection, he is faced with the problem of how to interpret the different trees. The comparison of two decision trees is therefore an important issue as the user needs such a comparison in order to understand what has changed. We have proposed to provide him with a measure of correspondence between the two trees that allows him to judge if he can accept the changes or not. In this paper, we propose a proper similarity measure. In case of a low similarity value, the expert has evidence to start exploring the reason for this change. Often, he can find things in the data acquisition that might have resulted in some noise and might be fixed. Copyright © 2014 John Wiley &amp; Sons, Ltd.

A Multi-Relational Decision Tree Learning (MRDTL) Approach: A Survey

Data Mining is the process of extracting useful knowledge from large set of data. There are number of data mining techniques available to find hidden knowledge from huge set of Data. Among these techniques classification is one of the techniques to predict the class label for unknown data based on previously known class labeled dataset. Several classification techniques like decision tree induction, Naivy Bayes model, rough set approach, fuzzy set theory and neural network are used for pattern extraction. Now a day's most of the real world data stored in relational database but the decision tree induction method is used to find knowledge from flat data relations only, but can't discover pattern from relational database. So to extract multi-relational pattern from relational tables we use MRDTL approach. In real world Missing value problem are common in many data mining application. This paper provides survey of multi-relational decision tree learning algorithm to discover hidden multi-relational pattern from relational data sets and also includes some simple technique to deal with missing value. Keyword: Data Mining, Multi-relational Data Mining Framework, Multi-relational Decision Tree Learning (MRDTL), Relational Database, Missing Value

Hierarchical multi-label classification using local neural networks

Hierarchical multi-label classification is a complex classification task where the classes involved in the problem are hierarchically structured and each example may simultaneously belong to more than one class in each hierarchical level. In this paper, we extend our previous works, where we investigated a new local-based classification method that incrementally trains a multi-layer perceptron for each level of the classification hierarchy. Predictions made by a neural network in a given level are used as inputs to the neural network responsible for the prediction in the next level. We compare the proposed method with one state-of-the-art decision-tree induction method and two decision-tree induction methods, using several hierarchical multi-label classification datasets. We perform a thorough experimental analysis, showing that our method obtains competitive results to a robust global method regarding both precision and recall evaluation measures.