Small Decision Trees Research Articles

Learning Small Decision Trees with Few Outliers: A Parameterized Perspective

Decision trees is a fundamental tool in machine learning for representing, classifying, and generalizing data. It is desirable to construct ``small'' decision trees, by minimizing either the size (s) or the depth (d) of the decision tree (DT). Recently, the parameterized complexity of Decision Tree Learning has attracted a lot of attention. We consider a generalization of Decision Tree Learning where given a classification instance E and an integer t, the task is to find a ``small'' DT that disagrees with E in at most t examples. We consider two problems: DTSO and DTDO, where the goal is to construct a DT minimizing s and d, respectively. We first establish that both DTSO and DTDO are W[1]-hard when parameterized by s+y and d+y, respectively, where y is the maximum number of features in which two differently labeled examples can differ. We complement this result by showing that these problems become FPT if we include the parameter t. We also consider the kernelization complexity of these problems and establish several positive and negative results for both DTSO and DTDO.

Learning Small Decision Trees for Data of Low Rank-Width

We consider the NP-hard problem of finding a smallest decision tree representing a classification instance in terms of a partially defined Boolean function. Small decision trees are desirable to provide an interpretable model for the given data. We show that the problem is fixed-parameter tractable when parameterized by the rank-width of the incidence graph of the given classification instance. Our algorithm proceeds by dynamic programming using an NLC decomposition obtained from a rank-width decomposition. The key to the algorithm is a succinct representation of partial solutions. This allows us to limit the space and time requirements for each dynamic programming step in terms of the parameter.

Design an Optimal Decision Tree based Algorithm to Improve Model Prediction Performance

Performance of decision trees is assessed by prediction accuracy for unobserved occurrences. In order to generate optimised decision trees with high classification accuracy and smaller decision trees, this study will pre-process the data. In this study, some decision tree components are addressed and enhanced. The algorithms should produce precise and ideal decision trees in order to increase prediction performance. Additionally, it hopes to create a decision tree algorithm with a tiny global footprint and excellent forecast accuracy. The typical decision tree-based technique was created for classification purposes and is used with various kinds of uncertain information. Prior to preparing the dataset for classification, the uncertain dataset was first processed through missing data treatment and other uncertainty handling procedures to produce the balanced dataset. Three different real-time datasets, including the Titanic dataset, the PIMA Indian Diabetes dataset, and datasets relating to heart disease, have been used to test the proposed algorithm. The suggested algorithm's performance has been assessed in terms of the precision, recall, f-measure, and accuracy metrics. The outcomes of suggested decision tree and the standard decision tree have been contrasted. On all three datasets, it was found that the decision tree with Gini impurity optimization performed remarkably well.

Robust Model Compression Using Deep Hypotheses

Machine Learning models should ideally be compact and robust. Compactness provides efficiency and comprehensibility whereas robustness provides stability. Both topics have been studied in recent years but in isolation. Here we present a robust model compression scheme which is independent of model types: it can compress ensembles, neural networks and other types of models into diverse types of small models. The main building block is the notion of depth derived from robust statistics. Originally, depth was introduced as a measure of the centrality of a point in a sample such that the median is the deepest point. This concept was extended to classification functions which makes it possible to define the depth of a hypothesis and the median hypothesis. Algorithms have been suggested to approximate the median but they have been limited to binary classification. In this study, we present a new algorithm, the Multiclass Empirical Median Optimization (MEMO) algorithm that finds a deep hypothesis in multi-class tasks, and prove its correctness. This led to our Compact Robust Estimated Median Belief Optimization (CREMBO) algorithm for robust model compression. We demonstrate the success of this algorithm empirically by compressing neural networks and random forests into small decision trees, which are interpretable models, and show that they are more accurate and robust than other comparable methods. In addition, our empirical study shows that our method outperforms Knowledge Distillation on DNN to DNN compression.

Parameterized Complexity of Small Decision Tree Learning

We study the NP-hard problem of learning a decision tree (DT) of smallest depth or size from data. We provide the first parameterized complexity analysis of the problem and draw a detailed parameterized complexity map for the natural parameters: size or depth of the DT, maximum domain size of all features, and the maximum Hamming distance between any two examples. Our main result shows that learning DTs of smallest depth or size is fixed-parameter tractable (FPT) parameterized by the combination of all three of these parameters. We contrast this FPT-result by various hardness results that underline the algorithmic significance of the considered parameters.

MC4.5 decision tree algorithm: an improved use of continuous attributes

C4.5 is one of the top ten data mining algorithms; it is the most widely used decision trees construction techniques. Although effective, it suffers from the problem of complexity when it deals with continuous attributes. It also leads to a certain level of information loss. Therefore, minimising such loss and reducing the time complexity is one of the main goals in this paper. With the intention of alleviating these problems, this paper presents a novel algorithm namely MC4.5, which proposes the statistical mean as an alternative to the C4.5 threshold selection process. To demonstrate the effectiveness of the new algorithm, a complete evaluation was launched to prove that MC4.5 complies with the objectives previously mentioned. From the theoretical perspective, we develop an analysis of the complexity to compare algorithms. Empirically, we conduct an experimental study using 30 datasets to prove that, in most cases, the proposed algorithm leads to smaller decision trees with better accuracy comparing to the C4.5 algorithm.

Serial, parallel and hierarchical decision making in primates.

The study of decision-making has mainly focused on isolated decisions where choices are associated with motor actions. However, problem-solving often involves considering a hierarchy of sub-decisions. In a recent study (Lorteije et al. 2015), we reported behavioral and neuronal evidence for hierarchical decision making in a task with a small decision tree. We observed a first phase of parallel evidence integration for multiple sub-decisions, followed by a phase in which the overall strategy formed. It has been suggested that a 'flat' competition between the ultimate motor actions might also explain these results. A reanalysis of the data does not support the critical predictions of flat models. We also examined the time-course of decision making in other, related tasks and report conditions where evidence integration for successive decisions is decoupled, which excludes flat models. We conclude that the flexibility of decision-making implies that the strategies are genuinely hierarchical.

IMPROVEMENT EVALUATION ON CERAMIC ROOF EXTRACTION USING WORLDVIEW-2 IMAGERY AND GEOGRAPHIC DATA MINING APPROACH

Advances in geotechnologies and in remote sensing have improved analysis of urban environments. The new sensors are increasingly suited to urban studies, due to the enhancement in spatial, spectral and radiometric resolutions. Urban environments present high heterogeneity, which cannot be tackled using pixel–based approaches on high resolution images. Geographic Object–Based Image Analysis (GEOBIA) has been consolidated as a methodology for urban land use and cover monitoring; however, classification of high resolution images is still troublesome. This study aims to assess the improvement on ceramic roof classification using WorldView-2 images due to the increase of 4 new bands besides the standard “Blue-Green-Red-Near Infrared” bands. Our methodology combines GEOBIA, C4.5 classification tree algorithm, Monte Carlo simulation and statistical tests for classification accuracy. Two samples groups were considered: 1) eight multispectral and panchromatic bands, and 2) four multispectral and panchromatic bands, representing previous high-resolution sensors. The C4.5 algorithm generates a decision tree that can be used for classification; smaller decision trees are closer to the semantic networks produced by experts on GEOBIA, while bigger trees, are not straightforward to implement manually, but are more accurate. The choice for a big or small tree relies on the user’s skills to implement it. This study aims to determine for what kind of user the addition of the 4 new bands might be beneficial: 1) the common user (smaller trees) or 2) a more skilled user with coding and/or data mining abilities (bigger trees). In overall the classification was improved by the addition of the four new bands for both types of users.

IMPROVEMENT EVALUATION ON CERAMIC ROOF EXTRACTION USING WORLDVIEW-2 IMAGERY AND GEOGRAPHIC DATA MINING APPROACH

Abstract. Advances in geotechnologies and in remote sensing have improved analysis of urban environments. The new sensors are increasingly suited to urban studies, due to the enhancement in spatial, spectral and radiometric resolutions. Urban environments present high heterogeneity, which cannot be tackled using pixel–based approaches on high resolution images. Geographic Object–Based Image Analysis (GEOBIA) has been consolidated as a methodology for urban land use and cover monitoring; however, classification of high resolution images is still troublesome. This study aims to assess the improvement on ceramic roof classification using WorldView-2 images due to the increase of 4 new bands besides the standard “Blue-Green-Red-Near Infrared” bands. Our methodology combines GEOBIA, C4.5 classification tree algorithm, Monte Carlo simulation and statistical tests for classification accuracy. Two samples groups were considered: 1) eight multispectral and panchromatic bands, and 2) four multispectral and panchromatic bands, representing previous high-resolution sensors. The C4.5 algorithm generates a decision tree that can be used for classification; smaller decision trees are closer to the semantic networks produced by experts on GEOBIA, while bigger trees, are not straightforward to implement manually, but are more accurate. The choice for a big or small tree relies on the user’s skills to implement it. This study aims to determine for what kind of user the addition of the 4 new bands might be beneficial: 1) the common user (smaller trees) or 2) a more skilled user with coding and/or data mining abilities (bigger trees). In overall the classification was improved by the addition of the four new bands for both types of users.

Learning invariants using decision trees and implication counterexamples

Inductive invariants can be robustly synthesized using a learning model where the teacher is a program verifier who instructs the learner through concrete program configurations, classified as positive, negative, and implications. We propose the first learning algorithms in this model with implication counter-examples that are based on machine learning techniques. In particular, we extend classical decision-tree learning algorithms in machine learning to handle implication samples, building new scalable ways to construct small decision trees using statistical measures. We also develop a decision-tree learning algorithm in this model that is guaranteed to converge to the right concept (invariant) if one exists. We implement the learners and an appropriate teacher, and show that the resulting invariant synthesis is efficient and convergent for a large suite of programs.

A Novel Neural Network Based Method Developed for Digit Recognition Applied to Automatic Speed Sign Recognition

This Paper presents a new hybrid technique for digit recognition applied to the speed limit sign recognition task. The complete recognition system consists in the detection and recognition of the speed signs in RGB images. A pretreatment is applied to extract the pictogram from a detected circular road sign, and then the task discussed in this work is employed to recognize digit candidates. To realize a compromise between performances, reduced execution time and optimized memory resources, the developed method is based on a conjoint use of a Neural Network and a Decision Tree. A simple Network is employed firstly to classify the extracted candidates into three classes and secondly a small Decision Tree is charged to determine the exact information. This combination is used to reduce the size of the Network as well as the memory resources utilization. The evaluation of the technique and the comparison with existent methods show the effectiveness.

Automatic sleep staging using state machine-controlled decision trees.

Automatic sleep staging from a reduced number of channels is desirable to save time, reduce costs and make sleep monitoring more accessible by providing home-based polysomnography. This paper introduces a novel algorithm for automatic scoring of sleep stages using a combination of small decision trees driven by a state machine. The algorithm uses two channels of EEG for feature extraction and has a state machine that selects a suitable decision tree for classification based on the prevailing sleep stage. Its performance has been evaluated using the complete dataset of 61 recordings from PhysioNet Sleep EDF Expanded database achieving an overall accuracy of 82% and 79% on training and test sets respectively. The algorithm has been developed with a very small number of decision tree nodes that are active at any given time making it suitable for use in resource-constrained wearable systems.

LEARNING HYPERPLANES THAT CAPTURES THE GEOMETRIC STRUCTURE OF CLASS REGIONS

Most of the decision tree algorithms rely on impurity measures to evaluate the goodness of hyperplanes at each node while learning a decision tree in a top-down fashion. These impurity measures are not differentiable with relation to the hyperplane parameters. Therefore the algorithms for decision tree learning using impurity measures need to use some search techniques for finding the best hyperplane at every node. These impurity measures don’t properly capture the geometric structures of the data. In this paper a Two-Class algorithm for learning oblique decision trees is proposed. Aggravated by this, the algorithm uses a strategy, to evaluate the hyperplanes in such a way that the (linear) geometric structure in the data is taken into consideration. At each node of the decision tree, algorithm finds the clustering hyperplanes for both the classes. The clustering hyperplanes are obtained by solving the generalized Eigen-value problem. Then the data is splitted based on angle bisector and recursively learn the left and right sub-trees of the node. Since, in general, there will be two angle bisectors; one is selected which is better based on an impurity measure gini index. Thus the algorithm combines the ideas of linear tendencies in data and purity of nodes to find better decision trees. This idea leads to small decision trees and better performance.

Online dynamic security assessment with missing pmu measurements: A data mining approach

A data mining approach using ensemble decision trees (DTs) learning is proposed for online dynamic security assessment (DSA), with the objective of mitigating the impact of possibly missing PMU data. Specifically, multiple small DTs are first trained offline using a random subspace method. In particular, the developed random subspace method exploits the hierarchy of wide-area monitoring system (WAMS), the locational information of attributes, and the availability of PMU measurements, so as to improve the overall robustness of the ensemble to missing data. Then, the performance of the trained small DTs is re-checked by using new cases in near real-time. In online DSA, viable small DTs are identified in case of missing PMU data, and a boosting algorithm is employed to quantify the voting weights of viable small DTs. The security classification decision for online DSA is obtained via a weighted voting of viable small DTs. A case study using the IEEE 39-bus system demonstrates the effectiveness of the proposed approach.

Geometric Decision Tree

In this paper, we present a new algorithm for learning oblique decision trees. Most of the current decision tree algorithms rely on impurity measures to assess the goodness of hyperplanes at each node while learning a decision tree in top-down fashion. These impurity measures do not properly capture the geometric structures in the data. Motivated by this, our algorithm uses a strategy for assessing the hyperplanes in such a way that the geometric structure in the data is taken into account. At each node of the decision tree, we find the clustering hyperplanes for both the classes and use their angle bisectors as the split rule at that node. We show through empirical studies that this idea leads to small decision trees and better performance. We also present some analysis to show that the angle bisectors of clustering hyperplanes that we use as the split rules at each node are solutions of an interesting optimization problem and hence argue that this is a principled method of learning a decision tree.

Secondary structure-based assignment of the protein structural classes

Structural class categorizes proteins based on the amount and arrangement of the constituent secondary structures. The knowledge of structural classes is applied in numerous important predictive tasks that address structural and functional features of proteins. We propose novel structural class assignment methods that use one-dimensional (1D) secondary structure as the input. The methods are designed based on a large set of low-identity sequences for which secondary structure is predicted from their sequence (PSSA(sc) model) or assigned based on their tertiary structure (SSA(sc)). The secondary structure is encoded using a comprehensive set of features describing count, content, and size of secondary structure segments, which are fed into a small decision tree that uses ten features to perform the assignment. The proposed models were compared against seven secondary structure-based and ten sequence-based structural class predictors. Using the 1D secondary structure, SSA(sc) and PSSA(sc) can assign proteins to the four main structural classes, while the existing secondary structure-based assignment methods can predict only three classes. Empirical evaluation shows that the proposed models are quite promising. Using the structure-based assignment performed in SCOP (structural classification of proteins) as the golden standard, the accuracy of SSA(sc) and PSSA(sc) equals 76 and 75%, respectively. We show that the use of the secondary structure predicted from the sequence as an input does not have a detrimental effect on the quality of structural class assignment when compared with using secondary structure derived from tertiary structure. Therefore, PSSA(sc) can be used to perform the automated assignment of structural classes based on the sequences.

Multiple classifier systems and layered learning based on CIDIM

In this paper we present two multiple classifier systems based on CIDIM. We also describe a layered learning approach that we have particularized using CIDIM as the basic classifier. CIDIM (Control of Induction by sample Division Method) is an algorithm that have been developed to induce accurate and small decision trees and to do this, it tries to reduce the overfitting using a local control of induction. The multiple classifier systems that we present (M-CIDIM and E-CIDIM) take advantage of the characteristics of CIDIM, but the approaches that have been developed can be extended to any other algorithm that shares the same characteristics.

The Impact of Simplification in a Sequential Rule-Based Model of Activity-Scheduling Behavior

The aim of this paper is to gain a better understanding of the impact of simplification on a sequential model of activity-scheduling behavior which uses feature-selection methods. To that effect, the predictive performance of the Albatross model, which incorporates nine different facets of activity–travel behavior, based on the original full decision trees, is compared with the performance of the model based on trimmed decision trees. The results indicate that significantly smaller decision trees can be used for modeling the different choice facets of the sequential model system without losing much in predictive power. The performance of the models is compared at three levels: the choice-facet level, the activity-pattern level (comparing the observed and generated sequences of activities), and the trip-matrix level, comparing the correlation coefficients that determine the strength of the associations between the observed and the predicted origin–destination matrices. The results indicate that the model based on the trimmed decision trees predicts activity-diary schedules with a minimum loss of accuracy at the decision level. Moreover, the results indicate a slightly better performance at the activity-pattern and the trip-matrix level.

Automatic sleep stage scoring based on waveform recognition method and decision‐tree learning

AbstractThere have been many trials in which the waveform recognition method, which is intended to replace expert observation by computer processing, has been used to extract the features of biological signals during sleep, and to automatically score sleep stages based on feature parameters. This paper proposes a waveform recognition method which extracts the feature parameters based on the characteristics of the biological signal during sleep, and a method of automatic sleep stage scoring by decision‐tree learning, which is currently considered to be one of the most successful machine learning methods in practice. As the first step in the method, the features corresponding to the state of the EEG, the EOG, and the EMG during sleep are compared to the features of characteristic waves such as the α‐wave, δ‐wave, sleep spindle, K‐complex, and REM, and the feature parameters needed in order to judge the sleep stage are extracted. Using canonical discriminant analysis and discretization method RWS based on the random walk, the feature parameters are converted to a small number of discrete variables. Based on training instances, obtained by the bootstrap method, a set of multiple small decision trees (a committee) is formed, and the sleep stage is scored by majority decision in the classification results. The method is applied to the PSG chart digital data provided by the Japan Sleep Society, and the performance of the system is evaluated experimentally. It is verified that the proposed method is promising as a method of automatic sleep stage scoring with high accuracy, requiring little expenditure of time in learning and classification. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(11): 1–13, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10248

TREE STRUCTURES OF LINEAR THRESHOLD UNITS FOR THE CLASSIFICATION OF NUMERIC EXAMPLES

A decision tree-based system for learning from numeric data is described. Results from linear algebra (pseudoinverse matrices) help the system to generate decision trees where the nodes are represented by linear threshold units minimizing the mean square error. The system's capability to provide good classifications with small decision trees is demonstrated on artificial and benchmark data.