Well-known C4 Research Articles

Multiple fairness criteria in decision tree learning

The use of algorithmic decision-making systems based on machine learning models has led to a need for fair (unbiased) and explainable classification outcomes. In particular, machine learning algorithms can encode biases, which might result in discriminatory decisions for certain groups such as gender, race, or age. Although a number of works on decision tree learning have been proposed to decrease the chance of discrimination, they usually focus on the use of a single fairness metric. In general, creating a model based on a single fairness metric is not a sufficient way to mitigate discrimination since bias can originate from various sources—e.g., the data itself or the optimization process. In this paper, we propose a novel decision tree learning process that utilizes multiple fairness metrics to address both group and individual discrimination. This is achieved by extending the attribute selection procedure to consider not only information gain but also gain in fairness. Computational experiments on fourteen different datasets with various sensitive features demonstrate that the proposed Fair-C4.5 models improve fairness without a loss in predictive accuracy when compared to the well-known C4.5 and the fairness-aware FFTree algorithms.

ProRE: An ACO- based programmer recommendation model to precisely manage software bugs

The process of assigning bugs to particular programmers is called bug assignment in software engineering. The programmer can fix the bugs by applying their knowledge. This research article presents an Ant colony optimization-based programmer recommendation model (ProRE) to manage software bugs precisely. The proposed ProRE model performs four operations: data pre-processing, i.e., data Pre-processing, extraction, feature selection, and programmer recommendation process. The feature selection stage utilized the Ant colony optimization (ACO) method to determine the appropriate subsets of features from all features. In the programmer recommendation stages, three programmer metrics, i.e., functionality ranking, bug occurrence, and mean Bug fixing time, are utilized for the recommendation assignment. The effectiveness of the proposed programmer recommendation system is assessed using datasets from Mozilla, Eclipse, Firefox, JBoss, and OpenFOAM. It is noted that the proposed model offers a better recommendation strategy over the other available systems. The simulation findings of the proposed ProRE model are also analyzed with well-known available ML methods, i.e., SVM, NB, and C4.5. It is observed that the recommendation results have improved by an average of 4%, 10%, and 12% compared to SVM, C4.5, and NB-based models. Programmer recommendation software is implemented for allocating the bugs to accurate programmers. It has been found that the proposed ProRE model generates more optimistic outcomes than existing ones.

Ensemble of classifier chains and Credal C4.5 for solving multi-label classification

In this work, we have considered the ensemble of classifier chains (ECC) algorithm in order to solve the multi-label classification (MLC) task. It starts from binary relevance algorithm (BR), a simple and direct approach to MLC that has been shown to provide good results in practice. Nevertheless, unlike BR, ECC aims to exploit the correlations between labels. ECC uses an algorithm of traditional supervised classification in order to approach the binary problems. Within this field, Credal C4.5 (CC4.5) is a new version of the well-known C4.5 algorithm that uses imprecise probabilities in order to estimate the probability distribution of the class variable. This new version of C4.5 algorithm has been shown to provide better performance when noisy datasets are classified. In MLC, the intrinsic noise might be higher than in traditional supervised classification. The reason is very simple: in MLC, there are multiple labels, whereas in traditional classification there is just a class variable. Thus, there is more probability of error for an instance. For the previous reasons, the performance of ECC with CC4.5 as base classifier is studied in this work. We have carried out an extensive experimental analysis with several multi-label datasets, different noise levels and a large number of evaluation metrics for MLC. This experimental study has shown that, generally, ECC has better performance with CC4.5 as base classifier than using C4.5. The higher is the label noise level introduced in the data, the more significative is this improvement. Therefore, it is probably suitable to use imprecise probabilities in Decision Trees within MLC.

Schema profiling of document-oriented databases

In document-oriented databases, schema is a soft concept and the documents in a collection can be stored using different local schemata. This gives designers and implementers augmented flexibility; however, it requires an extra effort to understand the rules that drove the use of alternative schemata when sets of documents with different —and possibly conflicting— schemata are to be analyzed or integrated. In this paper we propose a technique, called schema profiling, to explain the schema variants within a collection in document-oriented databases by capturing the hidden rules explaining the use of these variants. We express these rules in the form of a decision tree (schema profile). Consistently with the requirements we elicited from real users, we aim at creating explicative, precise, and concise schema profiles. The algorithm we adopt to this end is inspired by the well-known C4.5 classification algorithm and builds on two original features: the coupling of value-based and schema-based conditions within schema profiles, and the introduction of a novel measure of entropy to assess the quality of a schema profile. A set of experimental tests made on both synthetic and real datasets demonstrates the effectiveness and efficiency of our approach.

Machine learning techniques to discover genes with potential prognosis role in Alzheimer’s disease using different biological sources

Alzheimer’s disease is a complex progressive neurodegenerative brain disorder, being its prevalence expected to rise over the next decades. Unconventional strategies for elucidating the genetic mechanisms are necessary due to its polygenic nature. In this work, the input information sources are five: a public DNA microarray that measures expression levels of control and patient samples, repositories of known genes associated to Alzheimer’s disease, additional data, Gene Ontology and finally, a literature review or expert knowledge to validate the results. As methodology to identify genes highly related to this disease, we present the integration of three machine learning techniques: particularly, we have used decision trees, quantitative association rules and hierarchical cluster to analyze Alzheimer’s disease gene expression profiles to identify genes highly linked to this neurodegenerative disease, through changes in their expression levels between control and patient samples. We propose an ensemble of decision trees and quantitative association rules to find the most suitable configurations of the multi-objective evolutionary algorithm GarNet, in order to overcome the complex parametrization intrinsic to this type of algorithms. To fulfill this goal, GarNet has been executed using multiple configuration settings and the well-known C4.5 has been used to find the minimum accuracy to be satisfied. Then, GarNet is rerun to identify dependencies between genes and their expression levels, so we are able to distinguish between healthy individuals and Alzheimer’s patients using the configurations that overcome the minimum threshold of accuracy defined by C4.5 algorithm. Finally, a hierarchical cluster analysis has been used to validate the obtained gene-Alzheimer’s Disease associations provided by GarNet. The results have shown that the obtained rules were able to successfully characterize the underlying information, grouping relevant genes for Alzheimer Disease. The genes reported by our approach provided two well defined groups that perfectly divided the samples between healthy and Alzheimer’s Disease patients. To prove the relevance of the obtained results, a statistical test and gene expression fold-change were used. Furthermore, this relevance has been summarized in a volcano plot, showing two clearly separated and significant groups of genes that are up or down-regulated in Alzheimer’s Disease patients. A biological knowledge integration phase was performed based on the information fusion of systematic literature review, enrichment Gene Ontology terms for the described genes found in the hippocampus of patients. Finally, a validation phase with additional data and a permutation test is carried out, being the results consistent with previous studies.

Decentralised intelligent transport system with distributed intelligence based on classification techniques

This study is focused on a decentralised intelligent transportation systems with distributed intelligence based on classification techniques. The rationale behind this architecture is to offer a fully distributed, flexible and scalable system. The architecture encompasses the entire process of capture and management of available road data, enabling the generation of services to promote transportation efficiency. Besides that, thanks to the embedded classification techniques, the system is capable of predicting and reacting to certain events, facing them in an appropriate way. The aim of this work is to demonstrate how the system works in two different real-world use cases. To achieve this objective, how the architecture acts to deal with some incidences is proven. In addition, both use cases serve to show the effective communication between the different components of the system. Besides this, this work demonstrates the fundamental role played by the artificial intelligence techniques working in the system. The well-known C4.5 algorithm has been used for the accurate prediction of traffic congestion and pollution level. The authors explain in this work the reasons for using this classification technique, and the previous experiments performed.

A Data Classification Method for Inconsistency and Incompleteness Detection in Access Control Policy Sets

Access control policies may contain anomalies such as incompleteness and inconsistency, which can result in security vulnerabilities. Detecting such anomalies in large sets of complex policies automatically is a difficult and challenging problem. In this paper, we propose a novel method for detecting inconsistency and incompleteness in access control policies with the help of data classification tools well known in data mining. Our proposed method consists of three phases: firstly, we perform parsing on the policy data set; this includes ordering of attributes and normalization of Boolean expressions. Secondly, we generate decision trees with the help of our proposed algorithm, which is a modification of the well-known C4.5 algorithm. Thirdly, we execute our proposed anomaly detection algorithm on the resulting decision trees. The results of the anomaly detection algorithm are presented to the policy administrator who will take remediation measures. In contrast to other known policy validation methods, our method provides means for handling incompleteness, continuous values and complex Boolean expressions. In order to demonstrate the efficiency of our method in discovering inconsistencies, incompleteness and redundancies in access control policies, we also provide a proof-of-concept implementation.

데이터마이닝을 활용한 소프트웨어 개발인력의 업무 지속수행의도 결정요인 분석

According to the `2013 Performance Assessment Report on the Financial Program` from the National Assembly Budget Office, the unfilled recruitment ratio of Software(SW) Developers in South Korea was 25% in the 2012 fiscal year. Moreover, the unfilled recruitment ratio of highly-qualified SW developers reaches almost 80%. This phenomenon is intensified in small and medium enterprises consisting of less than 300 employees. Young job-seekers in South Korea are increasingly avoiding becoming a SW developer and even the current SW developers want to change careers, which hinders the national development of IT industries. The Korean government has recently realized the problem and implemented policies to foster young SW developers. Due to this effort, it has become easier to find young SW developers at the beginning-level. However, it is still hard to recruit highly-qualified SW developers for many IT companies. This is because in order to become a SW developing expert, having a long term experiences are important. Thus, improving job continuity intentions of current SW developers is more important than fostering new SW developers. Therefore, this study surveyed the job continuity intentions of SW developers and analyzed the factors associated with them. As a method, we carried out a survey from September 2014 to October 2014, which was targeted on 130 SW developers who were working in IT industries in South Korea. We gathered the demographic information and characteristics of the respondents, work environments of a SW industry, and social positions for SW developers. Afterward, a regression analysis and a decision tree method were performed to analyze the data. These two methods are widely used data mining techniques, which have explanation ability and are mutually complementary. We first performed a linear regression method to find the important factors assaociated with a job continuity intension of SW developers. The result showed that an `expected age` to work as a SW developer were the most significant factor associated with the job continuity intention. We supposed that the major cause of this phenomenon is the structural problem of IT industries in South Korea, which requires SW developers to change the work field from developing area to management as they are promoted. Also, a `motivation` to become a SW developer and a `personality (introverted tendency)` of a SW developer are highly importantly factors associated with the job continuity intention. Next, the decision tree method was performed to extract the characteristics of highly motivated developers and the low motivated ones. We used well-known C4.5 algorithm for decision tree analysis. The results showed that `motivation`, `personality`, and `expected age` were also important factors influencing the job continuity intentions, which was similar to the results of the regression analysis. In addition to that, the `ability to learn` new technology was a crucial factor for the decision rules of job continuity. In other words, a person with high ability to learn new technology tends to work as a SW developer for a longer period of time. The decision rule also showed that a `social position` of SW developers and a `prospect` of SW industry were minor factors influencing job continuity intensions. On the other hand, `type of an employment (regular position/ non-regular position)` and `type of company (ordering company/ service providing company)` did not affect the job continuity intension in both methods. In this research, we demonstrated the job continuity intentions of SW developers, who were actually working at IT companies in South Korea, and we analyzed the factors associated with them. These results can be used for human resource management in many IT companies when recruiting or fostering highly-qualified SW experts. It can also help to build SW developer fostering policy and to solve the problem of unfilled recruitment of SW Developers in South Korea.

HIERARCHICAL CLASSIFICATOR: A COGNITIVE APPROACH TO DECISION TREE BUILDING

We present a new algorithm that follows divide and conquer machine learning approach and exhibits a few interesting cognitive properties. The algorithm aims at building the decision tree with only one terminal node per class. Splits of tree nodes are constrained to functions that take identical values (true or false) for every instance within the same class. Appropriate splits are found through an exhaustive search in the attribute-value-based function space. Simple single-attribute functions are considered before complex multi-attribute k-DNF type ones. Redundant functions are also being incorporated into the decision tree. The unique structure of the decision tree results in that semantic interpretation can be attached to both terminal and non-terminal nodes, the task-specific set of classes is structured within a hierarchy of similarity relationships, and sources of recognition errors can be traced back (localized) to some particular function/split in the decision tree. The new algorithm was implemented, experimentally evaluated and compared with the well-known machine learning techniques Ripper and C4.5. Though limited in scope the experiments showed that the new algorithm can perform at least as well as Ripper and C4.5. Redundant knowledge incorporated into the decision tree helped to improve the recognition accuracy.

The use of vicinal-risk minimization for training decision trees

We propose the use of Vapnik's vicinal risk minimization (VRM) for training decision trees to approximately maximize decision margins. We implement VRM by propagating uncertainties in the input attributes into the labeling decisions. In this way, we perform a global regularization over the decision tree structure. During a training phase, a decision tree is constructed to minimize the total probability of misclassifying the labeled training examples, a process which approximately maximizes the margins of the resulting classifier. We perform the necessary minimization using an appropriate meta-heuristic (genetic programming) and present results over a range of synthetic and benchmark real datasets. We demonstrate the statistical superiority of VRM training over conventional empirical risk minimization (ERM) and the well-known C4.5 algorithm, for a range of synthetic and real datasets. We also conclude that there is no statistical difference between trees trained by ERM and using C4.5. Training with VRM is shown to be more stable and repeatable than by ERM.

Heuristic procedures for improving the predictability of a genetic programming financial forecasting algorithm

Financial forecasting is an important area in computational finance. Evolutionary Dynamic Data Investment Evaluator (EDDIE) is an established genetic programming (GP) financial forecasting algorithm, which has successfully been applied to a number of international financial datasets. The purpose of this paper is to further improve the algorithm’s predictive performance, by incorporating heuristics in the search. We propose the use of two heuristics: a sequential covering strategy to iteratively build a solution in combination with the GP search and the use of an entropy-based dynamic discretisation procedure of numeric values. To examine the effectiveness of the proposed improvements, we test the new EDDIE version (EDDIE 9) across 20 datasets and compare its predictive performance against three previous EDDIE algorithms. In addition, we also compare our new algorithm’s performance against C4.5 and RIPPER, two state-of-the-art classification algorithms. Results show that the introduction of heuristics is very successful, allowing the algorithm to outperform all previous EDDIE versions and the well-known C4.5 and RIPPER algorithms. Results also show that the algorithm is able to return significantly high rates of return across the majority of the datasets.

Particle Swarm Optimization Using Adaptive Boundary Correction for Human Activity Recognition

As a kind of personal lifelog data, activity data have been considered as one of the most compelling information to understand the users habits and to calibrate diagnoses. In this paper, we proposed a robust algorithm to sampling rates for human activity recognition, which identifies a user’s activity using accelerations from a triaxial accelerometer in a smartphone. Although a high sampling rate is required for high accuracy, it is not desirable for actual smartphone usage, battery consumption, or storage occupancy. Activity recognitions with well-known algorithms, including MLP, C4.5, or SVM, suffer from a loss of accuracy when a sampling rate of accelerometers decreases. Thus, we start from particle swarm optimization (PSO), which has relatively better tolerance to declines in sampling rates, and we propose PSO with an adaptive boundary correction (ABC) approach. PSO with ABC is tolerant of various sampling rate in that it identifies all data by adjusting the classification boundaries of each activity. The experimental results show that PSO with ABC has better tolerance to changes of sampling rates of an accelerometer than PSO without ABC and other methods. In particular, PSO with ABC is 6%, 25%, and 35% better than PSO without ABC for sitting, standing, and walking, respectively, at a sampling period of 32 seconds. PSO with ABC is the only algo-rithm that guarantees at least 80% accuracy for every activity at a sampling period of smaller than or equal to 8 seconds.

A fast and efficient multi-objective evolutionary learning scheme for fuzzy rule-based classifiers

During the last years, multi-objective evolutionary algorithms (MOEAs) have been extensively used to generate fuzzy rule-based systems characterized by different trade-offs between accuracy and complexity. In this paper, we propose an MOEA-based approach to learn concurrently the rule and data bases of fuzzy rule-based classifiers (FRBCs). In particular, the rule bases are generated by exploiting a rule and condition selection (RCS) strategy, which selects a reduced number of rules from a heuristically generated set of candidate rules and a reduced number of conditions for each selected rule during the evolutionary process. RCS can be considered as a rule learning in a constrained search space. As regards the data base learning, the membership function parameters of each linguistic term used in the rules are learned concurrently to the application of RCS. We tested our approach on twenty-four classification benchmarks and compared our results with the ones obtained by two similar state-of-the-art MOEA-based approaches and by two well-known non-evolutionary classification algorithms, namely FURIA and C4.5. Using non-parametric statistical tests, we show that our approach generates FRBCs with accuracy and complexity statistically comparable to, and sometimes better than, the ones generated by the two MOEA-based approaches, exploiting, however, only the 5% of the number of fitness evaluations used by these approaches. Further, the classifiers generated by our approach result to be more interpretable than the ones generated by the FURIA and C4.5 algorithms, while achieving the same accuracy level.

A new approach for discretizing continuous attributes in learning systems

Discretization is a process to convert continuous attributes into discrete format to represent signals for further data processing in learning systems. The main concern in discretization techniques is to find an optimal representation of continuous values with limited number of intervals that can effectively characterize the data and meanwhile minimize information loss. In this paper, we propose a novel class-attribute interdependency discretization algorithm (termed as NCAIC), which takes account of data distribution and the interdependency between all classes and attributes. In our proposed solution, the upper approximation of rough sets as a prime part of the discretization algorithm is applied, and the class-attribute mutual information is used to automatically control and adjust the scope of the discretization of continuous attributes. Some experiments with comparison to five other discretization algorithms are reported, where 13 benchmarked datasets extracted from UCI database and the well-known C4.5 decision tree tool are employed in this study. Results demonstrate that in general our proposed algorithm outperforms other tested discretization algorithms in terms of classification performance.

An ant colony optimization based feature selection for web page classification.

The increased popularity of the web has caused the inclusion of huge amount of information to the web, and as a result of this explosive information growth, automated web page classification systems are needed to improve search engines' performance. Web pages have a large number of features such as HTML/XML tags, URLs, hyperlinks, and text contents that should be considered during an automated classification process. The aim of this study is to reduce the number of features to be used to improve runtime and accuracy of the classification of web pages. In this study, we used an ant colony optimization (ACO) algorithm to select the best features, and then we applied the well-known C4.5, naive Bayes, and k nearest neighbor classifiers to assign class labels to web pages. We used the WebKB and Conference datasets in our experiments, and we showed that using the ACO for feature selection improves both accuracy and runtime performance of classification. We also showed that the proposed ACO based algorithm can select better features with respect to the well-known information gain and chi square feature selection methods.

The linguistic attribute hierarchy and its optimisation for classification

A hierarchical approach to decomposing a high-dimensional model into series of low-dimensional sub-models can be an effective way to overcome the `curse of dimensionality' problem. We investigate a hierarchy of linguistic decision trees (LDTs) for classification, and present linguistic interpretation of the hierarchy of LDTs. Due to the uncertain and non-linear relationship between input attributes and a goal, different hierarchies could have different performance for classification. We develop a GA with the linguistic ID3 in wrapper to optimize linguistic attribute hierarchy. The experimental results show that optimised linguistic attribute hierarchies perform better on the benchmark databases than a single LDT does, and they can greatly reduce the number of rules when the relationship between a goal variable and input attributes is highly uncertain and nonlinear. Comparing with well-known machine learning approaches, C4.5, Naive Bayes, and Neural Networks, the optimised linguistic attribute hierarchy achieves the highest accuracies for most tested databases. The trained hierarchy can be a real-time classifier if the optimization of hierarchies is performed offline.

Reusable components in decision tree induction algorithms

We propose a generic decision tree framework that supports reusable components design. The proposed generic decision tree framework consists of several sub-problems which were recognized by analyzing well-known decision tree induction algorithms, namely ID3, C4.5, CART, CHAID, QUEST, GUIDE, CRUISE, and CTREE. We identified reusable components in these algorithms as well as in several of their partial improvements that can be used as solutions for sub-problems in the generic decision tree framework. The identified components can now be used outside the algorithm they originate from. Combining reusable components allows the replication of original algorithms, their modification but also the creation of new decision tree induction algorithms. Every original algorithm can outperform other algorithms under specific conditions but can also perform poorly when these conditions change. Reusable components allow exchanging of solutions from various algorithms and fast design of new algorithms. We offer a generic framework for component-based algorithms design that enhances understanding, testing and usability of decision tree algorithm parts.

Discovering interesting knowledge from a science and technology database with a genetic algorithm

Data mining consists of extracting interesting knowledge from data. This paper addresses the discovery of knowledge in the form of prediction IF-THEN rules, which are a popular form of knowledge representation in data mining. In this context, we propose a genetic algorithm (GA) designed specifically to discover interesting fuzzy prediction rules. The GA searches for prediction rules that are interesting in the sense of being new and surprising for the user. This is done adapting a technique little exploited in the literature, which is based on user-defined general impressions (subjective knowledge). More precisely, a prediction rule is considered interesting (or surprising) to the extent that it represents knowledge that not only was previously unknown by the user but also contradicts his original believes. In addition, the use of fuzzy logic helps to improve the comprehensibility of the rules discovered by the GA. This is due to the use of linguistic terms that are natural for the user. A prototype was implemented and applied to a real-world science & technology database, containing data about the scientific production of researchers. The GA implemented in this prototype was evaluated by comparing it with the J4.8 algorithm, a variant of the well-known C4.5 algorithm. Experiments were carried out to evaluate both the predictive accuracy and the degree of interestingness (or surprisingness) of the rules discovered by both algorithms. The predictive accuracy obtained by the proposed GA was similar to the one obtained by J4.8, but the former, in general, discovered rules with fewer conditions. In addition it works with natural linguistic terms, which leads to the discovery of more comprehensible knowledge. The rules discovered by the proposed GA and the best rules discovered by J4.8 were shown to a user (a University Director) in an interview who evaluated the degree of interestingness (surprisingness) of the rules to him. In general the user considered the rules discovered by the GA much more interesting than the rules discovered by J4.8.

Comparing a genetic fuzzy and a neurofuzzy classifier for credit scoring

In this paper, we evaluate and contrast two types of fuzzy classifiers for credit scoring. The first classifier uses evolutionary optimization and boosting for learning fuzzy classification rules. The second classifier is a fuzzy neural network that employs a fuzzy variant of the classic backpropagation learning algorithm. The experiments are carried out on a real life credit scoring data set. It is shown that, for the case at hand, the boosted genetic fuzzy classifier performs better than both the neurofuzzy classifier and the well-known C4.5(rules) decision tree(rules) induction algorithm. However, the better performance of the genetic fuzzy classifier is offset by the fact that it infers approximate fuzzy rules which are less comprehensible for humans than the descriptive fuzzy rules inferred by the neurofuzzy classifier.

Improving Performance of the k-Nearest Neighbor Classifier by Combining Feature Selection with Feature Weighting.

The k-nearest neighbor (k-NN) classification is a simple and effective classification approach. However, it suffers from over-sensitivity problem due to irrelevant and noisy features. There are two ways to relax such sensitivity. One is to assign each feature a weight, and the other way is to select a subset of relevant features. Existing researches showed that both approaches can improve generalization accuracy, but it is impossible to predict which one is better for a specific dataset. In this paper, we propose an algorithm to improve the effectiveness of k-NN by combining these two approaches. Specifically, we select all relevant features firstly, and then assign a weight to each relevant feature. Experiments have been conducted on 14 datasets from the UCI Machine Learning Repository, and the results show that our algorithm achieves the highest accuracy or near to the highest accuracy on all test datasets. It increases generalization accuracy 8.68% on the average. It also achieves higher generalization accuracy compared with well-known machine learning algorithm IB1-4 and C4.5.