Cost-complexity Pruning Research Articles

The potential of decision trees as a tool to simplify broiler chicken welfare assessments

To simplify fast-growth broiler welfare assessments and use them as a benchmarking tool, decision trees were used to identify iceberg indicators discriminating flocks passing/failing welfare assessments as with the complete AWIN protocol. A dataset was constructed with data from 57 flocks and 3 previous projects. A final flock assessment score, previously not included in the dataset, was calculated and used as the benchmarking assessment classifier (pass/fail). A decision tree to classify flocks was built using the Chi-square Automatic Interaction Detection (CHAID) criterion. Cost-complexity pruning, and tenfold cross-validation were used. The final decision tree included cumulative mortality (%), immobile, lame birds (%), and birds with back wounds (%). Values were (mean ± se) 2.77 ± 0.14%, 0.16 ± 0.02%, 0.25 ± 0.02%, and 0.003 ± 0.001% for flocks passing the assessment; and 4.39 ± 0.49%, 0.24 ± 0.05%, 0.49 ± 0.09%, and 0.015 ± 0.006% for flocks failing. Cumulative mortality had the highest relative importance. The validated model correctly predicted 80.70% of benchmarking assessment outcomes. Model specificity was 0.8696; sensitivity was 0.5455. Decision trees can be useful to simplify welfare assessments. Model improvements will be possible as more information becomes available, and predictions are based on more samples.

How M5 Model Trees (M5-MT) on Continuous Data Are Used in Rainfall Prediction: An Experimental Evaluation

When using machine learning to predict a class with a continuous numeric value, there are several issues. Only a few machine-learning approaches are capable of doing so, but it remains one of the most difficult jobs to do. In this paper, we show how to use the M5 Model Tree, an approach that can handle continuous numeric data. This method is a stepwise procedure that employs linear functions at the leaf nodes of any created decision tree inducer (such as CART). These M5 model trees provide basic practical formulas such as standard deviation (SD), standard deviation reduction (SDR), cost-complexity pruning (CCP), and so on, which may be simply applied to different benchmark data by another user. This study examines the M5 Model Tree algorithm's capabilities for analysing rainfall data in the Kashmir portion of India's Union Territory of Jammu & Kashmir. One of the best suited models was the M5 model tree, which was built using (70–30) percent training and test ratios, respectively, and predicted an RMSE of 2.593, an MAE of 1.68, and a correlation coefficient (R2) of 0.478. Furthermore, M5 model trees produce models with a minimal number of trails, requiring less computing effort and making them more practical to use.

Numerical and Experimental Investigation of Meteorological Data Using Adaptive Linear M5 Model Tree for the Prediction of Rainfall

Predicting a class with a continuous numeric value encounters many problems when applying machine learning to the data. Only a few machine-learning techniques can do this, but it is still considered one of the most complex tasks to perform. In this study, we demonstrate one of the techniques called the M5 Model Tree, which can handle continuous numeric data. This technique is a stepwise algorithm and uses linear functions at the leaf nodes of any decision tree inducer (like CART) constructed. These M5 model trees generate simple practical formulas like standard deviation (SD), standard deviation reduction (SDR), cost-complexity pruning (CCP), etc., which can be easily applied by another user to some other benchmark data. This work assesses the abilities of the M5 Model Tree algorithm for the assessment of rainfall data across the Kashmir province of the Union Territory of Jammu & Kashmir, India. The construction of the M5 model tree developed using (70–30) % training and test ratio, respectively, was considered one of the best fit models, predicting an RMSE of 2.593, an MAE of 1.68, and a correlation coefficient (R2) of 0.478. Moreover, M5 model trees use a small number of trails to develop the models and thus need less computational time and are therefore more convenient to use.

Case-Ascertainment Models to Identify Adults with Obstructive Sleep Apnea Using Health Administrative Data: Internal and External Validation.

BackgroundThere is limited evidence on whether obstructive sleep apnea (OSA) can be accurately identified using health administrative data.Study Design and MethodsWe derived and validated a case-ascertainment model to identify OSA using linked provincial health administrative and clinical data from all consecutive adults who underwent a diagnostic sleep study (index date) at two large academic centers (Ontario, Canada) from 2007 to 2017. The presence of moderate/severe OSA (an apnea–hypopnea index≥15) was defined using clinical data. Of 39 candidate health administrative variables considered, 32 were tested. We used classification and regression tree (CART) methods to identify the most parsimonious models via cost-complexity pruning. Identified variables were also used to create parsimonious logistic regression models. All individuals with an estimated probability of 0.5 or greater using the predictive models were classified as having OSA.ResultsThe case-ascertainment models were derived and validated internally through bootstrapping on 5099 individuals from one center (33% moderate/severe OSA) and validated externally on 13,486 adults from the other (45% moderate/severe OSA). On the external cohort, parsimonious models demonstrated c-statistics of 0.75–0.81, sensitivities of 59–60%, specificities of 87–88%, positive predictive values of 79%, negative predictive values of 73%, positive likelihood ratios (+LRs) of 4.5–5.0 and –LRs of 0.5. Logistic models performed better than CART models (mean integrated calibration indices of 0.02–0.03 and 0.06–0.12, respectively). The best model included: sex, age, and hypertension at the index date, as well as an outpatient specialty physician visit for OSA, a repeated sleep study, and a positive airway pressure treatment claim within 1 year since the index date.InterpretationAmong adults who underwent a sleep study, case-ascertainment models for identifying moderate/severe OSA using health administrative data had relatively low sensitivity but high specificity and good discriminative ability. These findings could help study trends and outcomes of OSA individuals using routinely collected health care data.

Pruning trees in C-fuzzy random forest

Pruning decision trees is the way to decrease their size in order to reduce classification time and improve (or at least maintain) classification accuracy. In this paper, the idea of applying different pruning methods to C-fuzzy decision trees and Cluster–context fuzzy decision trees in C-fuzzy random forest is presented. C-fuzzy random forest is a classifier which we created and we are improving. This solution is based on fuzzy random forest and uses C-fuzzy decision trees or Cluster–context fuzzy decision trees—depending on the variant. Five pruning methods were adjusted to mentioned kind of trees and examined: Reduced Error Pruning (REP), Pessimistic Error Pruning (PEP), Minimum Error Pruning (MEP), Critical Value Pruning (CVP) and Cost-Complexity Pruning. C-fuzzy random forests with unpruned trees and trees constructed using each of these pruning methods were created. The evaluation of created forests was performed on eleven discrete decision class datasets (forest with C-fuzzy decision trees) and two continuous decision class datasets (forest with Cluster–context fuzzy decision trees). The experiments on eleven different discrete decision class datasets and two continuous decision class datasets were performed to evaluate five implemented pruning methods. Our experiments show that pruning trees in C-fuzzy random forest in general reduce computation time and improve classification accuracy. Generalizing, the best classification accuracy improvement was achieved using CVP for discrete decision class problems and REP for continuous decision class datasets, but for each dataset different pruning methods work well. The method which pruned trees the most was PEP and the fastest one was MEP. However, there is no pruning method which fits the best for all datasets—the pruning method should be chosen individually according to the given problem. There are also situations where it is better to remain trees unpruned.

Informative trees by visual pruning

The aim of this study is to provide visual pruning and decision tree selection for classification and regression trees. Specifically, we introduce an unedited tree graph to be made informative for recursive tree data partitioning. A decision tree is visually selected through a dendrogram-like procedure or through automatic tree-size selection. Our proposal is a one-step procedure whereby the most predictive paths are visualized. This method appears to be useful in all real world cases where tree-path interpretation is crucial. Experimental evaluations using real world data sets are presented. The performance was very similar to Classification and Regression Trees (CART) benchmarking methodology, showing that our method is a valid alternative to the well-known method of cost-complexity pruning.

Comparative performance of regression tree and parametric classification of savannah woody cover on SPOT 6 NAOMI imagery

Earth observation data potentially can provide ecosystem state indicative data for use in nature conservation. Full realisation of this potential requires robust algorithms that can extract the required ecosystem attributes from imagery. Woody cover is an indicative ecosystem state variable in savannah wilderness areas, whose extraction poses a challenge in reproducing the characteristically discontinuous cover. In this work the performance of the more modern, nonparametric regression tree and Random Forest classifiers in mapping savannah woody cover is compared with the parametric, potentially highly accurate sub-pixel classification approach. Detailed field quantification of % woody cover in 1 ha plots yielded data for the training stage of the two decision tree-based nonparametric classifiers. SPOT 6 NAOMI imagery whose timing excluded grass from the spectral signature of non-senescent vegetation was used. In order to improve the capability to extract woody cover the 6 m resolution of the multispectral bands was enhanced to 1.5 m, using high pass filter pan-sharpening. For the decision tree-based classifiers multiresolution image segmentation was then employed as preliminary step in extracting the woody cover. Objects approximating the 1 ha field plots were delineated at the coordinates of the respective field sampling sites. The univariate decision tree training stage used 60% of the sampling sites. The woody cover classes 0–19%, 20–39%, 40–59% and 60–79% were the labels of recursive partitioning terminal tree leaves. For Random Forest, up to 500 decision trees (Ntree) were specified. For regression tree modelling the 10-fold cross validation approach using the 1-SE rule was used for optimal tree selection, and pruning using 20% of the field sampling sites was accomplished using the cost complexity pruning technique. The remaining 20% of the field sampling sites were reserved for accuracy assessment, for which an additional stratified random sample of coordinates verified using a concurrent high resolution QuickBird image was used. Spectral signatures approximating Gaussian distributions were generated using woody end member sites for sub-pixel classification. Random Forest produced the highest overall accuracy (95.7%), followed by regression tree modelling (90.6%), and sub-pixel classification (73.3%). The two nonparametric classifiers correctly modelled the landscape distribution, occasional continuous formations and discontinuities in the woody cover. Based on the results, nonparametric classifiers appeared more appropriate for mapping the discontinuous savannah woody cover.

Power Quality Disturbances Feature Selection and Recognition Using Optimal Multi-Resolution Fast S-Transform and CART Algorithm

In order to improve the recognition accuracy and efficiency of power quality disturbances (PQD) in microgrids, a novel PQD feature selection and recognition method based on optimal multi-resolution fast S-transform (OMFST) and classification and regression tree (CART) algorithm is proposed. Firstly, OMFST is carried out according to the frequency domain characteristic of disturbance signal, and 67 features are extracted by time-frequency analysis to construct the original feature set. Subsequently, the optimal feature subset is determined by Gini importance and sorted according to an embedded feature selection method based on the Gini index. Finally, one standard error rule subtree evaluation methods were applied for cost complexity pruning. After pruning, the optimal decision tree (ODT) is obtained for PQD classification. The experiments show that the new method can effectively improve the classification efficiency and accuracy with feature selection step. Simultaneously, the ODT can be constructed automatically according to the ability of feature classification. In different noise environments, the classification accuracy of the new method is higher than the method based on probabilistic neural network, extreme learning machine, and support vector machine.

Interactive Post Mining Association Rules using Cost Complexity Pruning and Ontologies KDD

In data mining, association rule mining is very strong and limited by the huge amount of delivered rules, because of these so many problems facing to implementation. To overcome these drawbacks, several methods were proposed in the literature such as item sets concise, redundancy reduction, and post processing. Based on statistical information by using these methods the extracted rules may not be useful for the user. Thus, it is crucial to help the decision-controller with an efficient post processing step in order to reduce the number of rules. In this paper we have implemented a new interactive approach using cost complexity pruning and filter discovered rules, by using this approach we have to reduce the tree size with minimum number of error of validation set.

Adaptive partitioning schemes for bipartite ranking

Recursive partitioning methods are among the most popular techniques in machine learning. The purpose of this paper is to investigate how to adapt this methodology to the bipartite ranking problem. Following in the footsteps of the TreeRank approach developed in Clémençon and Vayatis (Proceedings of the 2008 Conference on Algorithmic Learning Theory, 2008 and IEEE Trans. Inf. Theory 55(9):4316–4336, 2009), we present tree-structured algorithms designed for learning to rank instances based on classification data. The main contributions of the present work are the following: the practical implementation of the TreeRank algorithm, well-founded solutions to the crucial issues related to the splitting rule and the choice of the “right” size for the ranking tree. From the angle embraced in this paper, splitting is viewed as a cost-sensitive classification task with data-dependent cost. Hence, up to straightforward modifications, any classification algorithm may serve as a splitting rule. Also, we propose to implement a cost-complexity pruning method after the growing stage in order to produce a “right-sized” ranking sub-tree with large AUC. In particular, performance bounds are established for pruning schemes inspired by recent work on nonparametric model selection. Eventually, we propose indicators for variable importance and variable dependence, plus various simulation studies illustrating the potential of our method.

Modeling Phone Duration of Lithuanian by Classification and Regression Trees, using Very Large Speech Corpus

Classification and regression tree approach was used in this research to model phone duration of Lithuanian. 300 thousand samples of vowels and 400 thousand samples of consonants extracted from VDU-AB20 corpus were used in experimental part of research. Set of 15 parameters characterizing phone and its context were selected for duration prediction. The most significant of them were: identifier (ID) of phone being predicted, adjacent phones IDs and number of phones in syllable. Models were built using two different data sets: one speaker and 20 speakers. The influence of cost complexity pruning and different values of pre pruning were investigated. Prediction by average leaf duration vs. prediction by median leaf duration was also compared. Investigation of most vivid errors was performed, speech rate normalization and trivial noise reduction were applied and influence on models evaluation parameters discussed. The achieved results, correlation 0.8 and 0.75 respectively for vowels and consonants, and RMSE of ~18 ms are comparable with those reported for Check, Hindi and Telugu, Korean.

FBP: A Frontier-Based Tree-Pruning Algorithm

A frontier-based tree-pruning algorithm (FBP) is proposed. The new method has an order of computational complexity comparable to cost-complexity pruning (CCP). Regarding tree pruning, it provides a full spectrum of information: namely, (1) given the value of the penalization parameter λ, it gives the decision tree specified by the complexity-penalization approach; (2) given the size of a decision tree, it provides the range of the penalization parameter λ, within which the complexity-penalization approach renders this tree size; (3) it finds the tree sizes that are inadmissible—no matter what the value of the penalty parameter is, the resulting tree based on a complexity-penalization framework will never have these sizes. Simulations on real data sets reveal a “surprise:” in the complexity-penalization approach, most of the tree sizes are inadmissible. FBP facilitates a more faithful implementation of cross validation (CV), which is favored by simulations. Using FBP, a stability analysis of CV is proposed.

Maximum a posteriori pruning on decision trees and its application to bootstrap BUMPing

The cost-complexity pruning generates nested subtrees and selects the best one. However, its computational cost is large since it uses holdout sample or cross-validation. On the other hand, the pruning algorithms based on posterior calculations such as BIC (MDL) and MEP are faster, but they sometimes produce too big or small trees to yield poor generalization errors. In this paper, we propose an alternative pruning procedure which combines the ideas of the cost-complexity pruning and posterior calculation. The proposed algorithm uses only training samples, so that its computational cost is almost same as the other posterior-based algorithms, and at the same time yields similar accuracies as the cost-complexity pruning. Moreover it can be used for comparing non-nested trees, which is necessary for the BUMPing procedure. The empirical results show that the proposed algorithm performs similarly as the cost-complexity pruning in standard situations and works better for BUMPing.

A Dynamic Programming Based Pruning Method for Decision Trees

This paper concerns a decision-tree pruning method, a key issue in the development of decision trees. We propose a new method that applies the classical optimization technique, dynamic programming, to a decision-tree pruning procedure. We show that the proposed method generates a sequence of pruned trees that are optimal with respect to tree size. The dynamic-programming-based pruning (DPP) algorithm is then compared with cost-complexity pruning (CCP) in an experimental study. The results of our study indicate that DPP performs better than CCP in terms of classification accuracy.

Monotone Shrinkage of Trees

We investigate a new method for regression trees which obtains estimates and predictions subject to constraints on the coefficients representing the effects of splits in the tree. The procedure leads to both shrinking of the node estimates and pruning of branches in the tree and for some problems gives better predictions than cost-complexity pruning used in the classification and regression tree (CART) algorithm. The new method is based on the least absolute shrinkage and selection operator (LASSO) method developed by Tibshirani.