Tree Induction Algorithm Research Articles

Increasing trust in AI through privacy preservation and model explainability: Federated Learning of Fuzzy Regression Trees

Federated Learning (FL) lets multiple data owners collaborate in training a global model without any violation of data privacy, which is a crucial requirement for enhancing users’ trust in Artificial Intelligence (AI) systems. Despite the significant momentum recently gained by the FL paradigm, most of the existing approaches in the field neglect another key pillar for the trustworthiness of AI systems, namely explainability. In this paper, we propose a novel approach for FL of fuzzy regression trees (FRTs), which are generally acknowledged as highly interpretable by-design models. The proposed FL procedure is designed for the scenario of horizontally partitioned data and is based on the transmission of aggregated statistics from the clients to a central server for the tree induction procedure. It is shown that the proposed approach faithfully approximates the ideal case in which the tree induction algorithm is applied on the union of all local datasets, while still ensuring privacy preservation. Furthermore, the FL approach brings benefits, in terms of generalization capability, compared to the local learning setting in which each participant learns its own FRT based only on the private, local, dataset. The adoption of linear models in the leaf nodes ensures a competitive level of performance, as assessed by an extensive experimental campaign on benchmark datasets. The analysis of the results covers both the aspects of accuracy and interpretability of FRT. Finally, we discuss the application of the proposed federated FRT to the task of Quality of Experience forecasting in an automotive case-study.

Enhancing Groundwater Quality Evaluation Using Associative Rule Mining Technique with Random Forest Split Gini Indexing Algorithm for Nitrate Concentration Analysis

Human actions and changing weather patterns are contributing to the growing demand for groundwater resources. Nevertheless, evaluating the quality of groundwater is crucial. Nitrate is a significant water contaminant that can lead to blue-baby syndrome or methemoglobinemia. Therefore, it is necessary to assess the level of nitrate in groundwater. Current methods involve evaluating the quality of groundwater and integrating it into the models. The inappropriate datasets, lack of performance, and other constraints are limitations of current methods. Ground water dataset is used and pre-processed the data’s. Selected data’s are feature extracted and associated with the rule ranking. In the suggested model, the use of associative rule mining technique has been implemented to address these challenges and assess nitrate levels in groundwater. The method of rule ranking is carried out using association rule mining technique to divide the datasets. The split gini indexing algorithm is introduced in the proposed model for data classification. The Split Gini Indexing algorithm is a decision tree induction algorithm that is used to build decision trees for classification tasks. It is based on the Gini impurity measure, which measures the heterogeneity of a dataset. The quality of groundwater has been classified using Naïve Bayes, SVM, and KNN algorithms. The proposed approach's efficiency is evaluated by calculating performance metrics such as precision, accuracy, F1-score, and recall values. The suggested method in the current research attains an improved accuracy of 0.99, demonstrating enhanced performance.

Generative Model for Decision Trees

Decision trees are among the most popular supervised models due to their interpretability and knowledge representation resembling human reasoning. Commonly-used decision tree induction algorithms are based on greedy top-down strategies. Although these approaches are known to be an efficient heuristic, the resulting trees are only locally optimal and tend to have overly complex structures. On the other hand, optimal decision tree algorithms attempt to create an entire decision tree at once to achieve global optimality. We place our proposal between these approaches by designing a generative model for decision trees. Our method first learns a latent decision tree space through a variational architecture using pre-trained decision tree models. Then, it adopts a genetic procedure to explore such latent space to find a compact decision tree with good predictive performance. We compare our proposal against classical tree induction methods, optimal approaches, and ensemble models. The results show that our proposal can generate accurate and shallow, i.e., interpretable, decision trees.

Better trees: an empirical study on hyperparameter tuning of classification decision tree induction algorithms

Better trees: an empirical study on hyperparameter tuning of classification decision tree induction algorithms

Big data decision tree for continuous-valued attributes based on unbalanced cut points

The decision tree is a widely used decision support model, which can quickly mine effective decision rules based on the dataset. The decision tree induction algorithm for continuous-valued attributes, based on unbalanced cut points, is efficient for mining decision rules; however, extending it to big data remains an unresolved. In this paper, two solutions are proposed to solve this problem: the first one is based on partitioning instance subsets, whereas the second one uses partitioning attribute subsets. The crucial of these two solutions is how to find the global optimal cut point from the set of local optimal cut points. For the first solution, the calculation of the Gini index of the cut points between computing nodes and the selection of the global optimal cut point by communication between these computing nodes is proposed. However, in the second solution, the division of the big data into subsets using attribute subsets in a way that all cut points of an attribute are on the same map node is proposed, the local optimal cut points can be found in this map node, then the global optimal cut point can be obtained by summarizing all local optimal cut points in the reduce node. Finally, the proposed solutions are implemented with two big data platforms, Hadoop and Spark, and compared with three related algorithms on four datasets. Experimental results show that the proposed algorithms can not only effectively solve the scalability problem, but also have lowest running time, the fastest speed and the highest efficiency under the premise of preserving the classification performance.

Multi-dimensional Bayesian network classifiers for partial label ranking

The label ranking problem consists in learning preference models from training datasets labeled with (possibly incomplete) rankings of the class labels. The goal is then to predict a ranking for a given unlabeled instance. This work focuses on a more general interpretation where both the training dataset and the prediction given as output allow tied class labels, i.e., there is no particular preference between them. This problem is known as the partial label ranking problem. This paper tackles the partial label ranking problem by transforming the ranking with ties into a set of discrete variables representing the preference relations (ranked ahead, tied with, and ranked behind) between each pair of class labels. The posterior probabilities for each pair are then used to fill the values of a preference matrix. This preference matrix is the basis for solving the rank aggregation problem required to obtain the output ranking with ties. This paper aims to exploit the resemblance of this problem with multi-label and multi-dimensional classification by studying the use of Bayesian network classifiers to compute the posterior probabilities for the new class structure, i.e., pairs of class labels. In particular, binary relevance with naive Bayes and averaged one-dependence estimators between the new class structure are used to solve the partial label ranking problem. Furthermore, bivariate relationships between all the pairs of class labels are considered. However, the complexity of the model grows significantly, which makes it necessary to reduce the number of allowed bivariate relationships between pairs. Thus, a feature selection method is included to select the more relevant subset of bivariate relationships. The experimental evaluation shows that our proposals are competitive in accuracy with the current instance-based and decision tree induction algorithms. Moreover, they outperform the existing mixture-based probabilistic graphical models, while the algorithms proposed are much faster.

A pivot-based simulated annealing algorithm to determine oblique splits for decision tree induction

We describe a new simulated annealing algorithm to compute near-optimal oblique splits in the context of decision tree induction. The algorithm can be interpreted as a walk on the cells of a hyperplane arrangement defined by the observations in the training set. The cells of this hyperplane arrangement correspond to subsets of oblique splits that divide the feature space in the same manner and the vertices of this arrangement reveal multiple neighboring solutions. We use a pivoting strategy to iterate over the vertices and to explore this neighborhood. Embedding this neighborhood search in a simulated annealing framework allows to escape local minima and increases the probability of finding global optimal solutions. To overcome the problems related to degeneracy, we rely on a lexicographic pivoting scheme. Our experimental results indicate that our approach is well-suited for inducing small and accurate decision trees and capable of outperforming existing univariate and oblique decision tree induction algorithms. Furthermore, oblique decision trees obtained with this method are competitive with other popular prediction models.

Regularized impurity reduction: accurate decision trees with complexity guarantees

Decision trees are popular classification models, providing high accuracy and intuitive explanations. However, as the tree size grows the model interpretability deteriorates. Traditional tree-induction algorithms, such as C4.5 and CART, rely on impurity-reduction functions that promote the discriminative power of each split. Thus, although these traditional methods are accurate in practice, there has been no theoretical guarantee that they will produce small trees. In this paper, we justify the use of a general family of impurity functions, including the popular functions of entropy and Gini-index, in scenarios where small trees are desirable, by showing that a simple enhancement can equip them with complexity guarantees. We consider a general setting, where objects to be classified are drawn from an arbitrary probability distribution, classification can be binary or multi-class, and splitting tests are associated with non-uniform costs. As a measure of tree complexity, we adopt the expected cost to classify an object drawn from the input distribution, which, in the uniform-cost case, is the expected number of tests. We propose a tree-induction algorithm that gives a logarithmic approximation guarantee on the tree complexity. This approximation factor is tight up to a constant factor under mild assumptions. The algorithm recursively selects a test that maximizes a greedy criterion defined as a weighted sum of three components. The first two components encourage the selection of tests that improve the balance and the cost-efficiency of the tree, respectively, while the third impurity-reduction component encourages the selection of more discriminative tests. As shown in our empirical evaluation, compared to the original heuristics, the enhanced algorithms strike an excellent balance between predictive accuracy and tree complexity.

Towards a Better Diagnosis of Prostate Cancer: Application of Machine Learning Algorithms

Introduction: Prostate cancer is one of the leading causes of death in men, and the early detection of this disease can be a significant factor in controlling and managing it. Applying data mining techniques can lead to the extraction of hidden knowledge from a huge amount of data and can help diagnose this disease by physicians. This study aims to determine the algorithm with the best performance to diagnose prostate cancer.Methods: In this study, nine data mining techniques, including Support Vector Machine, Decision Tree, Naive Bayes, K-Nearest Neighbors, Neural Network, Random Forest, Deep Learning, Auto-MLP, and Rule Induction algorithms, were used to extract hidden patterns from prostate cancer data. In this study, the data of 100 patients, which included eight characteristics, were used, and the RapidMiner Studio environment was employed for modeling. To compare the performance of the mentioned approaches used in this study to diagnose prostate cancer, accuracy, recall, precision, AUC, sensitivity, and specificity were calculated and reported for all techniques.Results: The results of this study showed that the accuracy of the applied algorithms was between 77% and 84%. Using different criteria to evaluate the techniques used showed that the two algorithms K-Nearest Neighbors and Neural Network, had better performance and accuracy (84%) than other methods. The sensitivity in these two algorithms was 80% for Neural Networks and 85% for K-Nearest Neighbors, respectively.Conclusion: The usage of different data mining techniques can lead to the discovery of hidden patterns among an enormous amount of data related to prostate cancer, and as a result, it leads to the early diagnosis of this disease and saves the subsequent costs.

Deep preference learning for multiple criteria decision analysis

We propose preference learning algorithms for inferring the parameters of a threshold-based sorting model from large sets of assignment examples. The introduced framework is adjusted to different scores originally used in Multiple Criteria Decision Analysis (MCDA). They include Ordered Weighted Average, an additive value function, the Choquet integral, a distance from the ideal and anti-ideal alternatives, and Net Flow Scores built on the results of outranking-based pairwise comparisons. As a concrete application of these models, we use Artificial Neural Networks with up to five hidden layers. Their components and architecture are designed to ensure high interpretability, which supports the models’ acceptance by domain experts. To learn the most favorable values of all parameters at once, we use a variant of a gradient descent optimization algorithm called AdamW. In this way, we make the MCDA methods suitable for handling vast, inconsistent information. The extensive experiments on various benchmark problems indicate that the introduced algorithms are competitive in predictive accuracy quantified in terms of Area Under Curve and the 0/1 loss. In this regard, some approaches outperform the state-of-the-art algorithms, including generalizations of logistic regression, mathematical programming, rule ensemble and tree induction algorithms, or dedicated heuristics.

Induction of decision trees as classification models through metaheuristics

• The three types of metaheuristic and the hyperheuristic-based methods to build decision tree induction algorithms are included, not just evolutionary-algorithm-based approaches for decision tree induction. • Three types of implementation strategies are introduced, according to the place where a metaheuristict is used in the decision tree induction process. • The differences in the solution representation are highlighted, as they impact the metaheuristic-based implementation. • The principal components of each method are detailed, such as fitness measures and variation operators. • An analysis of the experimental studies performed in these methods is conducted. The induction of decision trees is a widely-used approach to build classification models that guarantee high performance and expressiveness. Since a recursive-partitioning strategy guided for some splitting criterion is commonly used to induce these classifiers, overfitting, attribute selection bias, and instability to small training set changes are well-known problems in them. Other approaches, such as incremental induction, classifier ensembles, and the global search in the decision-tree-space, have been implemented to overcome these problems. In particular, metaheuristics such as simulated annealing, genetic algorithms, genetic programming, and ant colony optimization have been used to induce compact and accurate decision trees. This paper presents a state-of-the-art review of the use of single-solution-based metaheuristics and swarm and evolutionary computation algorithms to build decision trees as classification models. We outline the decision-tree-induction process components and detail the existing literature studies on metaheuristic-based approaches to building these classifiers. Several timelines showing the chronological order in which these approaches were introduced in the literature are included. A summary analysis of these studies is also conducted, focusing on their internal components and experimental studies. This work provides a useful reference point for future research in this field.

An integrated knowledge-based system for early detection of eye refractive error using data mining

Refractive error is one of optical defect in the human visual system. Refractive error is a very common disease these days in all populations and in all age groups. Uncorrected and undetected refractive error contributes to visual impairment, blindness and places a considerable burden on a person in the world. The long use of technological devices such as smart phones also poses a new burden on the human eye. The intensity and brightness of these digital devices open a new door for high prevalence of eye refractive errors. Early medical diagnosis of the disease may help in avoiding complications and blindness. Data mining algorithms can be applied to help in ophthalmology and detection of an eye disease at an early stage. So mining the ophthalmology data in efficient manner is a critical issue. This research work deals with development of an integrated knowledge-based system that helps to detect eye refractive error early and provides appropriate advice for the patients. In this study, the hybrid knowledge discovery process model of data mining that was developed for academic research is used. About 9000 ophthalmology data from selected eye health centers are used to build the model. The sample data was preprocessed for missing values, outliers, and noise. Then the model is built using decision tree (J48 and REPTree) and rule induction (JRip and part) algorithms. The part algorithm has registered better predictive performance with accuracy of 60% and 96.45% for subjective and objective based model evaluation, respectively as compared to J48, REPTree, and JRip. Finally, the knowledge discovered with this algorithm is further used to build the knowledge-based systems. The Java programing language is used to integrate data mining results to knowledge-based system. The performance of the proposed system is evaluated by preparing test cases. Overall, the knowledge based system resulted in 89.2% accuracy. Finally the study concludes that discovering knowledge using data mining techniques could be used as a functional eye refractive error detection system.

Predicting and interpreting oxide glass properties by machine learning using large datasets

With the advent of powerful computer simulation techniques, it is time to move from the widely used knowledge-guided empirical methods to approaches driven by data science, mainly machine learning algorithms. We investigated the predictive performance of three machine learning algorithms for six different glass properties. For such, we used an extensive dataset of about 150,000 oxide glasses, which was segmented into smaller datasets for each property investigated. Using the decision tree induction, k-nearest neighbors, and random forest algorithms, selected from a previous study of six algorithms, we induced predictive models for glass transition temperature, liquidus temperature, elastic modulus, thermal expansion coefficient, refractive index, and Abbe number. Moreover, each model was induced with default and tuned hyperparameter values. We demonstrate that, apart from the elastic modulus (which had the smallest training dataset), the induced predictive models for the other five properties yield a comparable uncertainty to the usual data spread. However, for glasses with extremely low or high values of these properties, the prediction uncertainty is significantly higher. Finally, as expected, glasses containing chemical elements that are poorly represented in the training set yielded higher prediction errors. The method developed here calls attention to the success and possible pitfalls of machine learning algorithms. The analysis of the SHAP values indicated the key elements that increase or decrease the value of the modeled properties. It also estimated the maximum possible increase or decrease. Insights gained by this analysis can help empirical compositional tuning and computer-aided inverse design of glass formulations.

HYPER HEURISTIC EVOLUTIONARY APPROACH FOR CONSTRUCTING DECISION TREE CLASSIFIERS

Decision tree models have earned a special status in predictive modeling since these are considered comprehensible for human analysis and insight. Classification and Regression Tree (CART) algorithm is one of the renowned decision tree induction algorithms to address the classification as well as regression problems. Finding optimal values for the hyper parameters of a decision tree construction algorithm is a challenging issue. While making an effective decision tree classifier with high accuracy and comprehensibility, we need to address the question of setting optimal values for its hyper parameters like the maximum size of the tree, the minimum number of instances required in a node for inducing a split, node splitting criterion and the amount of pruning. The hyper parameter setting influences the performance of the decision tree model. As researchers, we know that no single setting of hyper parameters works equally well for different datasets. A particular setting that gives an optimal decision tree for one dataset may produce a sub-optimal decision tree model for another dataset. In this paper, we present a hyper heuristic approach for tuning the hyper parameters of Recursive and Partition Trees (rpart), which is a typical implementation of CART in statistical and data analytics package R. We employ an evolutionary algorithm as hyper heuristic for tuning the hyper parameters of the decision tree classifier. The approach is named as Hyper heuristic Evolutionary Approach with Recursive and Partition Trees (HEARpart). The proposed approach is validated on 30 datasets. It is statistically proved that HEARpart performs significantly better than WEKA’s J48 algorithm in terms of error rate, F-measure, and tree size. Further, the suggested hyper heuristic algorithm constructs significantly comprehensible models as compared to WEKA’s J48, CART and other similar decision tree construction strategies. The results show that the accuracy achieved by the hyper heuristic approach is slightly less as compared to the other comparative approaches.

GEOBIA, TREE DECISION AND HIERARCHICAL CLASSIFICATION FOR MAPPING GULLY EROSION

The gullies provoke environmental, social and financial damages. The application of corrective and preventive measures needs gullies mapping and monitoring. In this scope, this study proposes a methodology for gullies delimitation using object-oriented image analysis. For such, there were used high spatial resolution imagery and ALS data applied for two study areas, one in Uberlandia-Minas Gerais (Brazil) and another one in Queensland (Australia). The objects were generated by multiresolution segmentation. The most important attributes on the delimitation of the gullies were selected using decision tree induction algorithms, being them: spectral, altimetric and texture. Classifications by decision trees and hierarchical were carried out. The use of decision tree allowed the selection of attributes and the establishment of preliminary decision rules. However, since this procedure did not use fuzzy logic, mixtures between classes could not be evidenced in the rule base. Moreover, the classification was performed by a factor of scale only, which did not allow the identification of all the constituent features of the gully. In hierarchical classification, the procedure is performed on different scales, allowing the use of fuzzy logic to describe different degrees of membership in each class, which makes it a very attractive method for cases such as this study, where there is mixing of classes. The classification obtained with hierarchical classification it was more reliable with the field truth, by allowing the use of different scales, uncertainty insert and integration of knowledge, compared to the automatic classification by decision tree.

Hard-ODT: Hardware-Friendly Online Decision Tree Learning Algorithm and System

Decision trees are machine learning models commonly used in various application scenarios. In the era of big data, traditional decision tree induction algorithms are not suitable for learning large-scale datasets due to their stringent data storage requirement. Online decision tree learning algorithms have been devised to tackle this problem by concurrently training with incoming samples and providing inference results. However, even the most up-to-date online tree learning algorithms still suffer from either high memory usage or high computational intensity with dependency and long latency, making them challenging to implement in hardware. To overcome these difficulties, we introduce a new quantile-based algorithm to improve the induction of the Hoeffding tree, one of the state-of-the-art online learning models. The proposed algorithm is lightweight in terms of both memory and computational demand, while still maintaining high generalization ability. A series of optimization techniques dedicated to the proposed algorithm have been investigated from the hardware perspective, including coarse-grained and fine-grained parallelism, dynamic and memory-based resource sharing, pipelining with data forwarding. Following this, we present Hard-ODT, a high-performance, hardware-efficient and scalable online decision tree learning system on a field-programmable gate array (FPGA) with system-level optimization techniques. Performance and resource utilization are modeled for the complete learning system for early and fast analysis of the tradeoff between various design metrics. Finally, we propose a design flow in which the proposed learning system is applied to FPGA run-time power monitoring as a case study. Experimental results show that our proposed algorithm outperforms the state-of-the-art Hoeffding tree learning method, leading to 0.05% to 12.3% improvement in inference accuracy. Real implementation of the complete learning system on the FPGA demonstrates a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$384\times $ </tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1581\times $ </tex-math></inline-formula> speedup in execution time over the state-of-the-art design. The power modeling strategy with Hard-ODT achieves an average power prediction error within 4.93% of a commercial gate-level power estimation tool.

Spam Detection in Social Media Networking Sites using Ensemble Methodology with Cross Validation

Social media networking sites are more popular over Internet. The Internet users spend more amount of time on social media sites like Twitter, Facebook, Instagram and LinkedIn etc. The social media networking users share their ideas, opinions, information and make new friends. Social networking sites provide large amount of valuable information to the users. This large amount of information in social media attracts spammers to misuse information. These spammers create fake accounts and spread irrelevant information to the genuine users. The spam message information may be advertisements, malicious links to disturb the natural users. This spam data in social media is a very serious problem. Spam detection in social media networking sites is critical process. To extract spam messages in social media various spam detection methodologies are developed by researchers. In this paper we proposed an ensemble methodology for identification spam on Twitter social media network. In this methodology we used Decision tree induction algorithm, Naïve bayes algorithm and KNN algorithm to construct a model. As part of this approach, we compare the classification results of any two classification algorithms, if both classifiers predict the same result, then we finalize the class of tweet under investigation. If the predicted classes of both classification algorithms differ, then we use the prediction of third algorithm as the final class label of tweet. To measure the performance of our model we used precision, recall and F measure.

Decision tree classifier: a detailed survey

Decision tree classifier (DTC) is one of the well-known methods for data classification. The most significant feature of DTC is its ability to change the complicated decision making problems into simple processes, thus finding a solution which is understandable and easier to interpret. This paper provides a brief review on various algorithms developed in literature for constructing and representing decision trees, splitting criteria for selecting best attribute and pruning methods. The readers will be able to understand why decision trees are more popular among all other methods of classification, what are their uses, limitations and applications in different diverse areas. They will also come to know about a decision tree induction algorithms, splitting criteria, pruning methods, concepts of ensemble methods, fuzzy decision trees, hybridisation of DTCs, etc. These enhancements are found very helpful in solving complex datasets with less computation in very short time period while achieving high accuracy.

Hardware Acceleration of Sparse Oblique Decision Trees for Edge Computing

This paper presents a hardware accelerator for sparse decision trees intended for FPGA applications. To the best of authors’ knowledge, this is the first accelerator of this type. Beside the hardware accelerator itself, a novel algorithm for induction of sparse decision trees is also presented. Sparse decision trees can be attractive because they require less memory resources and can be more efficiently processed using specialized hardware compared to traditional oblique decision trees. This can be of significant interest, particularly, in the edge-based applications, where memory and compute resources as well as power consumption are severely constrained. The performance of the proposed sparse decision tree induction algorithm as well as developed hardware accelerator are studied using standard benchmark datasets obtained from the UCI Machine Learning Repository database. The results of the experimental study indicate that the proposed algorithm and hardware accelerator are very favourably compared with some of the existing solutions.

Inquiry of Personal Intelligence Of Adolescence and Early Adulthood College Students of Tamilnadu using Id3 Algorithm

Educational Data Mining is an Emerging research domain which focus on extracting knowledge from educational databases to promote the learning environment. Educational psychology helps to understand the differences in learning process of a person from cognitive and behavioral perspective. This research aims to predict the personal intelligence of adolescence and early adulthood who are studying under graduate and post graduate courses in educational institution. Adolescence plays a very import role in the development of personality to new dimensions in human life. The Adolescence physical, mental, social, moral and spiritual outlooks undergo revolutionary changes. Many teachers and parents fail to asses these changes and they do not like to slacken their control over them. Psychologist has stressed to properly channelize the behavior of adolescence and give them adequate education. In this paper machine learning technique with decision tree induction algorithm was used to analyze the personal intelligence of college students. To construct the decision tree, Entropy and Information Gain are used as attribute selection measures in ID3 algorithm. Applying data mining techniques in the field of Educational Psychology is a new method and the rules generated from the decision tree helps to identify the personal intelligence of the students. This result helps the Educators to improve the learning environment better for Adolescence.