Size Of Decision Tree Research Articles

XClusters: Explainability-First Clustering

We study the problem of explainability-first clustering where explainability becomes a first-class citizen for clustering. Previous clustering approaches use decision trees for explanation, but only after the clustering is completed. In contrast, our approach is to perform clustering and decision tree training holistically where the decision tree's performance and size also influence the clustering results. We assume the attributes for clustering and explaining are distinct, although this is not necessary. We observe that our problem is a monotonic optimization where the objective function is a difference of monotonic functions. We then propose an efficient branch-and-bound algorithm for finding the best parameters that lead to a balance of clustering accuracy and decision tree explainability. Our experiments show that our method can improve the explainability of any clustering that fits in our framework.

Z-number-valued rule-based decision trees

As a novel architecture of a fuzzy decision tree constructed on fuzzy rules, the fuzzy rule-based decision tree (FRDT) achieved better performance in terms of both classification accuracy and the size of the resulted decision tree than other classical decision trees such as C4.5, LADtree, BFtree, SimpleCart and NBTree. The concept of Z-number extends the classical fuzzy number to model both uncertain and partial reliable information. Z-numbers have significant potential in rule-based systems due to their strong representation capability. This paper designs a Z-number-valued rule-based decision tree (ZRDT) and provides the learning algorithm. Firstly, the information gain is used to replace the fuzzy confidence in FRDT to select features in each rule. Additionally, we use the negative samples to generate the second fuzzy numbers that adjust the first fuzzy numbers and improve the model's fit to the training data. The proposed ZRDT is compared with the FRDT with three different parameter values and two classical decision trees, PUBLIC and C4.5, and a decision tree ensemble method, AdaBoost.NC, in terms of classification effect and size of decision trees. Based on statistical tests, the proposed ZRDT has the highest classification performance with the smallest size for the produced decision tree.

Topological Forest

We propose a new ML model called Topological Forest that contains an ensemble of decision trees. Unlike a vanilla Random Forest, Topological Forest has a special training process that selects a smaller number of decision trees on a topological graph representation that TDA Mapper constructs. Compared to Vanilla Random Forest, Topological Forest significantly improves the computational efficiency of inference time due to the smaller ensemble size and selection of better decision trees while keeping the diversity of decision trees. Our experiments show that Topological Forest can speed up inference time by more than 100x on average while compromising at most 2% reduction in the AUC metric for the prediction quality.

PHeavy: Predicting Heavy Flows in the Programmable Data Plane

Since heavy flows account for a significant fraction of network traffic, being able to predict heavy flows has benefited many network management applications for mitigating link congestion, scheduling of network capacity, exposing network attacks and so on. Existing machine learning based predictors are largely implemented on the control plane of Software Defined Networking (SDN) paradigm. As a result, frequent communication between the control and data planes can cause unnecessary overhead and additional delay in decision making. In this paper, we present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pHeavy</i> , a machine learning based scheme for predicting heavy flows directly on the programmable data plane, thus eliminating network overhead and latency to SDN controller. Considering the scarce memory and limited computation capability in the programmable data plane, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pHeavy</i> includes a packet processing pipeline which deploys pre-trained decision tree models for in-network prediction. We have implemented <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pHeavy</i> in both bmv2 software switch and P4 hardware switch (i.e., Barefoot Tofino). Evaluation results demonstrate that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pHeavy</i> has achieved 85% and 98% accuracy after receiving the first 5 and 20 packets of a flow respectively, while being able to reduce the size of decision tree by 5.4x on average. More importantly, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">pHeavy</i> can predict heavy flows at line rate on the P4 hardware switch.

Risk Factors for High-Risk Adenoma on the First Lifetime Colonoscopy Using Decision Tree Method: A Cross-Sectional Study in 6,047 Asymptomatic Koreans.

Background/Aims: As risk of colorectal neoplasm is varied even in persons with “average-risk,” risk evaluation and tailored screening are needed. This study aimed to evaluate the risk factors of high-risk adenoma (HRA) in healthy individuals and determine the characteristics of advanced neoplasia (AN) among individual polyps.Methods: Asymptomatic adults who underwent the first lifetime screening colonoscopy at the Seoul National University Hospital Healthcare System Gangnam Center (SNUH GC) were recruited from 2004 to 2007 as SNUH GC Cohort and were followed for 10 years. Demographic and clinical characteristics were compared between the subjects with and without AN (≥10 mm in size, villous component, and/or high-grade dysplasia and/or cancer) or HRA (AN and/or 3 or more adenomas). For individual polyps, correlations between clinical or endoscopic features and histologic grades were evaluated using a decision tree method.Results: A total of 6,047 subjects were included and 5,621 polyps were found in 2,604 (43%) subjects. Advanced age, male sex, and current smoking status were statistically significant with regards to AN and HRA. A lower incidence of AN was observed in subjects taking aspirin. In the decision tree model, the location, shape, and size of the polyp, and sex of the subject were key predictors of the pathologic type. A weak but significant association was observed between the prediction of the final tree and the histological grouping (Kendall's tau-c = 0.142, p < 0001).Conclusions: Advanced neoplasia and HRA can be predicted using several individual characteristics and decision tree models.

Improved algorithm of decision tree based on neural network

Decision trees have been applied to solve many data mining problems due to their superior learning and classification capabilities, and have achieved good results. However, for dealing with big data and complex model problems, decision trees show insufficient accuracy and overfitting. In order to solve these problems, neural network is introduced as a decision tree node, and an improved algorithm based on neural network decision tree is proposed. In the neural network decision tree, the classifier learning consists of two stages: the first stage uses a heuristic algorithm with reduced uncertainty to divide the big data, and stops the growth of the decision tree until the node dividing ability is below a certain threshold; in the second stage, the neural network is used to classify the decision-making leaf node with generalization ability. The experimental results show that compared with the traditional classification learning algorithm, the algorithm has a higher accuracy rate and it can determine the size of decision tree through structural adaptation for the classification problem of identifying big data and complex patterns.

Wildcard Fields-Based Partitioning for Fast and Scalable Packet Classification in Vehicle-to-Everything.

Vehicle-to-Everything (V2X) requires high-speed communication and high-level security. However, as the number of connected devices increases exponentially, communication networks are suffering from huge traffic and various security issues. It is well known that performance and security of network equipment significantly depends on the packet classification algorithm because it is one of the most fundamental packet processing functions. Thus, the algorithm should run fast even with the huge set of packet processing rules. Unfortunately, previous packet classification algorithms have focused on the processing speed only, failing to be scalable with the rule-set size. In this paper, we propose a new packet classification approach balancing classification speed and scalability. It can be applied to most decision tree-based packet classification algorithms such as HyperCuts and EffiCuts. It determines partitioning fields considering the rule duplication explicitly, which makes the algorithm memory-effective. In addition, the proposed approach reduces the decision tree size substantially with the minimal sacrifice of classification performance. As a result, we can attain high-speed packet classification and scalability simultaneously, which is very essential for latest services such as V2X and Internet-of-Things (IoT).

Sparse alternating decision tree

Alternating decision tree (ADTree) is a special decision tree representation that brings interpretability to boosting, a well-established ensemble algorithm. This has found success in wide applications. However, existing variants of ADTree are implementing univariate decision nodes where potential interactions between features are ignored. To date, there has been no multivariate ADTree. We propose a sparse version of multivariate ADTree such that it remains comprehensible. The proposed sparse ADTree is empirically tested on UCI datasets as well as spectral datasets from the University of Eastern Finland (UEF). We show that sparse ADTree is competitive against both univariate decision trees (original ADTree, C4.5, and CART) and multivariate decision trees (Fisher's decision tree and a single multivariate decision tree from oblique Random Forest). It achieves the best average rank in terms of prediction accuracy, second in terms of decision tree size and faster induction time than existing ADTree. In addition, it performs especially well on datasets with correlated features such as UEF spectral datasets. Thus, the proposed sparse ADTree extends the applicability of ADTree to a wider variety of applications.

Decisions: Algebra, Implementation, and First Experiments

Classification is a constitutive part in many different fields of Computer Science. There exist several approaches that capture and manipulate classification information in order to construct a specific classification model. These approaches are often tightly coupled to certain learning strategies, special data structures for capturing the models, and to how common problems, e.g. fragmentation, replication and model overfitting, are addressed. In order to unify these different classification approaches, we define a Decision Algebra which defines models for classification as higher order decision functions abstracting from their implementations using decision trees (or similar), decision rules, decision tables, etc. Decision Algebra defines operations for learning, applying, storing, merging, approximating, and manipulating models for classification, along with some general algebraic laws regardless of the implementation used. The Decision Algebra abstraction has several advantages. First, several useful Decision Algebra operations (e.g., learning and deciding) can be derived based on the implementation of a few core operations (including merging and approximating). Second, applications using classification can be defined regardless of the different approaches. Third, certain properties of Decision Algebra operations can be proved regardless of the actual implementation. For instance, we show that the merger of a series of probably accurate decision functions is even more accurate, which can be exploited for efficient and general online learning. As a proof of the Decision Algebra concept, we compare decision trees with decision graphs, an efficient implementation of the Decision Algebra core operations, which capture classification models in a non-redundant way. Compared to classical decision tree implementations, decision graphs are 20% faster in learning and classification without accuracy loss and reduce memory consumption by 44%. This is the result of experiments on a number of standard benchmark data sets comparing accuracy, access time, and size of decision graphs and trees as constructed by the standard C4.5 algorithm. Finally, in order to test our hypothesis about increased accuracy when merging decision functions, we merged a series of decision graphs constructed over the data sets. The result shows that on each step the accuracy of the merged decision graph increases with the final accuracy growth of up to 16%.

Decision Tree Generation Algorithm without Pruning

On the generation of decision tree based on rough set model, for the sake of classification accuracy, existing algorithms usually partition examples too specific. And it is hard to avoid the negative impact caused by few special examples on decision tree. In order to obtain this priority in traditional decision tree algorithm based on rough set, the sample is partitioned much more meticulously. Inevitably, a few exceptional samples have negative effect on decision tree. And this leads that the generated decision tree seems too large to be understood. It also reduces the ability in classifying and predicting the coming data. To settle these problems, the restrained factor is introduced in this paper. For expanding nodes in generating decision tree algorithm, besides traditional terminating condition, an additional terminating condition is involved when the restrained factor of sample is higher than a given threshold, then the node will not be expanded any more. Thus, the problem of much more meticulous partition is avoided. Furthermore, the size of decision tree generated with restrained factor involved will not seem too large to be understood.

부스팅 트리에서 적정 트리사이즈의 선택에 관한 연구

범주형 목표변수를 잘 예측하기 위한 데이터마이닝 방법 중에서 최근에는 여러 단일 분류자를 결합한 앙상블 기법이 많이 활용되고 있다. 앙상블 기법 가운데 부스팅은 재표본 시 분류하기 어려운 관찰치의 가중치를 높여 분류자가 해당 관찰치에 보다 집중할 수 있도록 함으로써 다른 앙상블 기법에 비해 오차를 효과적으로 감소시키는 방법으로 알려져 있다. 부스팅을 구성하는 분류자를 의사결정나무로 둔 부스팅 트리 모형의 경우 각 트리의 사이즈를 결정해야 하는데, 본 연구에서는 자료 별로 부스팅 트리에 가장 적합한 트리사이즈가 서로 다를수 있다고 가정하고, 주어진 자료에 맞는 트리사이즈를 추정하는 문제에 대해 논의하였다. 우선 트리사이즈가 부스팅 트리의 정확도에 중요한 영향을 미치는가를 파악하기 위하여 28개의 자료를 대상으로 실험을 수행하였으며, 그 결과 트리사이즈를 결정하는 문제가 모형 전체의 성능을 결정하는데 상당한 역할을 한다는 것을 확인할 수 있었다. 또한 그 결과를 바탕으로 최적의 트리사이즈에 영향을 미칠 것으로 판단되는 몇 가지 특성 변수를 정의하고, 해당 변수를 이용하여 부스팅 트리에서의 최적 트리사이즈를 설명하는 모형을 구성해 보았다. 자료 별로 고유한 최적의 트리사이즈는 자료의 특성에 의존적일 가능성도 있으므로 본 연구에서 제안하는 추정방법은 최적 트리사이즈를 결정하기 위한 출발점 또는 가이드라인으로 활용하는 것이 적절할 것이다. 기존에는 부스팅 트리의 사이즈에 대한 값으로 목표변수의 범주의 개수를 활용하였는데, 본 모형에서 제안하는 트리사이즈의 추정치로 부스팅 트리를 구축한 경우 기존방법에 비해 분류정확도를 유의미하게 개선하는 것을 확인할 수 있었다. This article is to find the right size of decision trees that performs better for boosting algorithm. First we defined the tree size D as the depth of a decision tree. Then we compared the performance of boosting algorithm with different tree sizes in the experiment. Although it is an usual practice to set the tree size in boosting algorithm to be small, we figured out that the choice of D has a significant influence on the performance of boosting algorithm. Furthermore, we found out that the tree size D need to be sufficiently large for some dataset. The experiment result shows that there exists an optimal D for each dataset and choosing the right size D is important in improving the performance of boosting. We also tried to find the model for estimating the right size D suitable for boosting algorithm, using variables that can explain the nature of a given dataset. The suggested model reveals that the optimal tree size D for a given dataset can be estimated by the error rate of stump tree, the number of classes, the depth of a single tree, and the gini impurity.

Improved direct product theorems for randomized query complexity

The “direct product problem” is a fundamental question in complexity theory which seeks to understand how the difficulty in computing a function on each of k independent inputs scales with k. We prove the following direct product theorem (DPT) for query complexity: if every T-query algorithm has success probability at most \({1 - \varepsilon}\) in computing the Boolean function f on input distribution μ, then for α ≤ 1, every \({\alpha \varepsilon Tk}\)-query algorithm has success probability at most \({(2^{\alpha \varepsilon}(1-\varepsilon))^k}\) in computing the k-fold direct product \({f^{\otimes k}}\) correctly on k independent inputs from μ. In light of examples due to Shaltiel, this statement gives an essentially optimal trade-off between the query bound and the error probability. Using this DPT, we show that for an absolute constant α > 0, the worst-case success probability of any α R 2(f) k-query randomized algorithm for \({f^{\otimes k}}\) falls exponentially with k. The best previous statement of this type, due to Klauck, Špalek, and de Wolf, required a query bound of O(bs(f) k). Our proof technique involves defining and analyzing a collection of martingales associated with an algorithm attempting to solve \({f^{\otimes k}}\). Our method is quite general and yields a new XOR lemma and threshold DPT for the query model, as well as DPTs for the query complexity of learning tasks, search problems, and tasks involving interaction with dynamic entities. We also give a version of our DPT in which decision tree size is the resource of interest.

New Decision Tree Algorithm with Restrained Factor Involved

Abstract It is the accuracy of classification that decides whether the algorithm is prior or not in generating decision tree. In order to obtain this priority in traditional decision tree algorithm based on rough set, the sample is partitioned much more meticulously. Inevitably, a few exceptional samples have negative effect on decision tree. And this leads that the generated decision tree seems too large to be understood. It also reduces the ability in classifying and predicting the coming data. To settle these problems, the restrained factor is introduced in this paper. For expanding nodes in generating decision tree algorithm, besides traditional terminating condition, an additional terminating condition is involved when the restrained factor of sample is higher than a given threshold, then the node will not be expanded any more. Thus, the problem of much more meticulous partition is avoided. Furthermore, the size of decision tree generated with restrained factor involved will not seem too large to be understood.

Consistency of randomized and finite sized decision tree ensembles

Regression via classification (RvC) is a method in which a regression problem is converted into a classification problem. A discretization process is used to covert continuous target value to classes. The discretized data can be used with classifiers as a classification problem. In this paper, we use a discretization method, Extreme Randomized Discretization, in which bin boundaries are created randomly to create ensembles. We present an ensemble method for RvC problems. We show theoretically for a set of problems that if the number of bins is three, the proposed ensembles for RvC perform better than RvC with the equal-width discretization method. We use these results to show that infinite-sized ensembles, consisting of finite-sized decision trees, created by a pure randomized method (split points are created randomly), are not consistent. We also theoretically show, using a set of regression problems, that the performance of these ensembles is dependent on the size of member decision trees.

Latent Attribute Space Tree Classifiers

提出一种潜在属性空间树分类器(latent attribute space tree classifier,简称LAST)框架,通过将原属性空间变换到更容易分离数据或更符合决策树分类特点的潜在属性空间,突破传统决策树算法的决策面局限,改善树分类器的泛化性能.在LAST 框架下,提出了两种奇异值分解斜决策树(SVD (singular value decomposition) oblique decision tree,简称SODT)算法,通过对全局或局部数据进行奇异值分解,构建正交的潜在属性空间,然后在潜在属性空间内构建传统的单变量决策树或树节点,从而间接获得原空间内近似最优的斜决策树.SODT 算法既能够处理整体数据与局部数据分布相同或不同的数据集,又可以充分利用有标签和无标签数据的结构信息,分类结果不受样本随机重排的影响,而且时间复杂度还与单变量决策树算法相同.在复杂数据集上的实验结果表明,与传统的单变量决策树算法和其他斜决策树算法相比,SODT 算法的分类准确率更高,构建的决策树大小更稳定,整体分类性能更鲁棒,决策树构建时间与C4.5 算法相近,而远小于其他斜决策树算法.;A framework of latent attribute space tree classifier (LAST) is proposed in this paper. LAST transforms data from the original attribute space into the latent attribute space, which is easier for data separation or more suitable for tree classifier, so that the decision boundary of the traditional decision tree can be extended and its generalization ability can be improved. This paper presents two SVD (singular value decomposition) oblique decision tree (SODT) algorithms based on the LAST framework. SODT first performs SVD on global and/or local data to construct orthogonal latent attribute space. Then, traditional decision tree or tree nodes are built in that space.Finally, SODT obtains the approximately optimal oblique decision tree of the original space. SODT can not only handle datasets with similar or different distribution between global and local data, but also can make full use of the structure information of the labelled and unlabelled data and produce the same classification results no matter how the observations are arranged. Besides, the time complexity of SODT is identical to that of the univariate decision tree. Experimental results show that compared with the traditional univariate decision tree algorithm C4.5 and the oblique decision tree algorithms OC1 and CART-LC, SODT gives higher classification accuracy, more stable decision tree size and comparable tree-construction time as C4.5, which is much less than that of OC1 and CART-LC.

Learning Monotone Decision Trees in Polynomial Time

We give an algorithm that learns any monotone Boolean function $\fisafunc$ to any constant accuracy, under the uniform distribution, in time polynomial in n and in the decision tree size of $f.$ This is the first algorithm that can learn arbitrary monotone Boolean functions to high accuracy, using random examples only, in time polynomial in a reasonable measure of the complexity of $f.$ A key ingredient of the result is a new bound showing that the average sensitivity of any monotone function computed by a decision tree of size s must be at most $\sqrt{\log s}$. This bound has proved to be of independent utility in the study of decision tree complexity [O. Schramm, R. O'Donnell, M. Saks, and R. Servedio, Every decision tree has an influential variable, in Proceedings of the 46th Annual IEEE Symposium on Foundations of Computer Science, IEEE Computer Society, Los Alamitos, CA, 2005, pp. 31–39]. We generalize the basic inequality and learning result described above in various ways—specifically, to partition size (a stronger complexity measure than decision tree size), p-biased measures over the Boolean cube (rather than just the uniform distribution), and real-valued (rather than just Boolean-valued) functions.

Finding the right decision tree's induction strategy for a hard real world problem

Decision trees have been already successfully used in medicine, but as in traditional statistics, some hard real world problems can not be solved successfully using the traditional way of induction. In our experiments we tested various methods for building univariate decision trees in order to find the best induction strategy. On a hard real world problem of the Orthopaedic fracture data with 2637 cases, described by 23 attributes and a decision with three possible values, we built decision trees with four classical approaches, one hybrid approach where we combined neural networks and decision trees, and with an evolutionary approach. The results show that all approaches had problems with either accuracy, sensitivity, or decision tree size. The comparison shows that the best compromise in hard real world problem decision trees building is the evolutionary approach.

FINDING SMALL EQUIVALENT DECISION TREES IS HARD

Two decision trees are called decision equivalent if they represent the same function, i.e., they yield the same result for every possible input. We prove that given a decision tree and a number, to decide if there is a decision equivalent decision tree of size at most that number is NP-complete. As a consequence, finding a decision tree of minimal size that is decision equivalent to a given decision tree is an NP-hard problem. This result differs from the well-known result of NP-hardness of finding a decision tree of minimal size that is consistent with a given training set. Instead our result is a basic result for decision trees, apart from the setting of inductive inference. On the other hand, this result differs from similar results for BDDs and OBDDs: since in decision trees no sharing is allowed, the notion of decision tree size is essentially different from BDD size.

Deterministic and Nondeterministic Decision Trees for Rough Computing

In the paper, infinite information systems are considered which are used in pattern recognition, discrete optimization, computational geometry. Depth and size of deterministic and nondeterministic decision trees over such information systems are studied. Two classes of infinite information systems are investigated. Systems from these classes are best from the point of view of time complexity and space complexity of deterministic as well as nondeterministic decision trees. In proofs methods of test theory [1] and rough set theory [6, 9] are used.

An exponential lower bound on the size of algebraic decision trees for Max

We prove an exponential lower bound on the size of any fixed degree algebraic decision tree for solving MAX, the problem of finding the maximum of n real numbers. This complements the n— 1 lower bound of [Rabin (1972)] on the depth of algebraic decision trees for this problem. The proof in fact gives an exponential lower bound on the size of the polyhedral decision problem MAX= for testing whether the j-th number is the maximum among a list of n real numbers. Previously, except for linear decision trees, no nontrivial lower bounds on the size of algebraic decision trees for any familiar problems are known. We also establish an interesting connection between our lower bound and the maximum number of minimal cutsets for any rank-d hypergraph on n vertices.