Learning Classification Rules Research Articles

A Consumer Behavior Analysis Framework toward Improving Market Performance Indicators: Saudi’s Retail Sector as a Case Study

Studying customer behavior and anticipating future trends is a challenging task, as customer behavior is complex and constantly evolving. To effectively anticipate future trends, businesses need to analyze large amounts of data, use sophisticated analytical techniques, and stay up-to-date with the latest research and industry trends. In this paper, we propose a comprehensive framework to identify trends in consumer behavior using multiple layers of processing, including clustering, classification, and association rule learning. The aim is to help a major retailer in Saudi Arabia better understand customer behavior by utilizing the power of big data analysis. The proposed framework is presented as being generalized to gain insight into the generated big data and enable data-driven decision-making in other relevant domains. We developed this framework in collaboration with a large supermarket chain in Saudi Arabia, which provided us with over 1,000,000 sales transaction records belonging to around 30,000 of their loyal customers. In this study, we apply our proposed framework to those data as a case study and present our initial results of consumer clustering and association rules for each cluster. Moreover, we analyze our findings to figure out how we can further utilize intelligence to predict customer behavior in clustered groups.

On the efficient implementation of classification rule learning

AbstractRule learning methods have a long history of active research in the machine learning community. They are not only a common choice in applications that demand human-interpretable classification models but have also been shown to achieve state-of-the-art performance when used in ensemble methods. Unfortunately, only little information can be found in the literature about the various implementation details that are crucial for the efficient induction of rule-based models. This work provides a detailed discussion of algorithmic concepts and approximations that enable applying rule learning techniques to large amounts of data. To demonstrate the advantages and limitations of these individual concepts in a series of experiments, we rely on BOOMER—a flexible and publicly available implementation for the efficient induction of gradient boosted single- or multi-label classification rules.

Differentiating Crohn’s disease from intestinal tuberculosis using a fusion correlation neural network

Crohn’s disease (CD) and intestinal tuberculosis (ITB) have many similarities and overlaps in their clinical manifestations and medical imaging features, which makes it difficult to accurately differentiate CD from ITB. Random forests have unique advantages in discovering specific indicators and measuring relevance. By fusing the attributes and the correlation information between the sample and training set, we propose a fusion correlation neural network (FCNN) to diagnose CD and ITB patients. A total of 287 cases with fifty-nine indicators were collected from the Second Xiangya Hospital of Central South University. An FCNN allows one to explain the feature selection and learning classification rules in neural networks. The experiments show that perianal fistulas, skin and mucosal lesions, intestinal perforation, comb sign, and cobblestone appearance tend to be associated with CD; and ascites show a higher association with intestinal tuberculosis. These conclusions are consistent with the experience of clinicians. Finally, the age, disease course, white blood cell, hemoglobin, platelets, C-reactive protein, and T-SPOT of patients were used to train the FCNN. The average accuracy of the FCNN exceeds 91%, showing better performance than existing linear regression standards and other machine learning models.

Binary classification rule generation from decomposed data

Learning classification rules from data that do not fit in the available memory is a challenging task. The goal of this study is to develop an approach for generating binary classification rules from decomposed data that are equivalent in terms of quality to those found over the whole data. In the proposed approach, each class is divided into the same arbitrary small number of subtables. For each pair of subsets from different classes, rule sets are induced using any sequential covering algorithm. Rule sets generated from the same positive class subset and different negative class subsets are merged using an operator constructed on the basis of Cartesian product and conjunction operators. The rule sets obtained in this way are joined into one set. During the rule merging, unnecessary rules are removed. It is proven that for training data, the quality of the rule set generated using the approach is the same as that for the whole data. It is experimentally verified that for test data, the quality of classification is comparable with that obtained using a nondecomposed data approach.

Exploiting skew-adaptive delimitation mechanism for learning expressive classification rules

The expressivity of machine learning algorithms is considered to be critical in intelligent data analysis tasks for practical application. As an alternative set of classification rule learning algorithms to conventional decision tree, Prism family of algorithms induce modular rules concisely, thus exhibiting good expressiveness for human users. However, existing Prism rule induction techniques are limited by the assumption of Gaussian distribution for quantitative attributes, and may not be available for real life data analyzing, in which skewness is commonly observed. For this reason, we investigate a skew-adaptive mechanism for rule term boundary delimitation in Prism inductive learning. The propose algorithm, called P2-Prism, could learn expressive classification rules directly from quantitative data beyond Gaussian distribution. By employing statistical inference characteristics of Poisson process, our mechanism provides a significant contribution to classification rule inductive learning with adaption of skewed data distribution. The experimental evaluation of our algorithm demonstrates its skew-adaptive superiority on benchmark datasets, comparing with state-of-the-art algorithms. Furthermore, it is shown that P2-Prism is a robust classifier in the presence of various levels of noise, which further reveals its adaptability to the skewness of data distribution.

Prediction of hepatocellular carcinoma patient survival using machine learning classification rules.

e15649 Background: The outcome prediction of hepatocellular carcinoma (HCC) is conventionally determined by evaluating tissue samples obtained during surgical removal of the primary tumor focusing on their clinical and pathologic features. Recently, accumulating evidence suggests that cancer development is comprehensively modulated by the host’s immune system underlying the importance of immunological biomarkers for the prediction of HCC prognosis. However, an integrated predictive algorism incorporating clinical characteristic and immune features still remain to be established. Methods: We obtained respectable stage II HCC specimens, along with adjacent para-tumor tissues from 221 patients who underwent surgical resection at Eastern Hepatobiliary Surgery Hospital, (Shanghai, China) from 2015 through April 2018. Characteristics such as CD8+, CD163+, tumor-infiltrating lymphocytes (TILs) were obtained for further model construction used to predict the status of 3 survival indexes: Overall Survival (OS ,≤ 24 or > 24 month), Progression Free Survival (PFS, ≤ 6 or > 6 month), and Recurrence/Death (RD). Mutual information and coefficient between each feature and the survival indexes were tested to remove low scoring features after data cleaning and standardization. Furthermore, recursive features selection was preformed to obtain the optimal features combination. Finally, supervised learning techniques include either boosting or bagging strategy were used to fit and predict model with a grid-search method optimizing the parameters. Meanwhile, a cross validation procedure with 0.2 proportion of test cohort was randomly carried out for 10 times to evaluate the model. Results: We finally confirmed 15 biomarkers from the 46 candidates as features for the survival status prediction by using a 221 patients cohort. Among them, the top 10 most important biomarkers, included both clinical and immune attributes. The AUC of our model for survival indexes (OS, PFS, RD) was ranged from 0.76 (RD) to 0.8 (PFS), and the accuracy was above 0.85. Conclusions: We describe the integrative analysis of the clinical and immune features which collectively contribute to the survival index of HCC. Machine learning techniques, such as Gradient Boosting and random forest classifier , have a great promise for using in HCC cancer survival prediction.

Do Additional Features Help or Hurt Category Learning? The Curse of Dimensionality in Human Learners.

The curse of dimensionality, which has been widely studied in statistics and machine learning, occurs when additional features cause the size of the feature space to grow so quickly that learning classification rules becomes increasingly difficult. How do people overcome the curse of dimensionality when acquiring real-world categories that have many different features? Here we investigate the possibility that the structure of categories can help. We show that when categories follow a family resemblance structure, people are unaffected by the presence of additional features in learning. However, when categories are based on a single feature, they fall prey to the curse, and having additional irrelevant features hurts performance. We compare and contrast these results to three different computational models to show that a model with limited computational capacity best captures human performance across almost all of the conditions in both experiments.

Tunable structure priors for Bayesian rule learning for knowledge integrated biomarker discovery.

AIMTo develop a framework to incorporate background domain knowledge into classification rule learning for knowledge discovery in biomedicine.METHODSBayesian rule learning (BRL) is a rule-based classifier that uses a greedy best-first search over a space of Bayesian belief-networks (BN) to find the optimal BN to explain the input dataset, and then infers classification rules from this BN. BRL uses a Bayesian score to evaluate the quality of BNs. In this paper, we extended the Bayesian score to include informative structure priors, which encodes our prior domain knowledge about the dataset. We call this extension of BRL as BRLp. The structure prior has a λ hyperparameter that allows the user to tune the degree of incorporation of the prior knowledge in the model learning process. We studied the effect of λ on model learning using a simulated dataset and a real-world lung cancer prognostic biomarker dataset, by measuring the degree of incorporation of our specified prior knowledge. We also monitored its effect on the model predictive performance. Finally, we compared BRLp to other state-of-the-art classifiers commonly used in biomedicine.RESULTSWe evaluated the degree of incorporation of prior knowledge into BRLp, with simulated data by measuring the Graph Edit Distance between the true data-generating model and the model learned by BRLp. We specified the true model using informative structure priors. We observed that by increasing the value of λ we were able to increase the influence of the specified structure priors on model learning. A large value of λ of BRLp caused it to return the true model. This also led to a gain in predictive performance measured by area under the receiver operator characteristic curve (AUC). We then obtained a publicly available real-world lung cancer prognostic biomarker dataset and specified a known biomarker from literature [the epidermal growth factor receptor (EGFR) gene]. We again observed that larger values of λ led to an increased incorporation of EGFR into the final BRLp model. This relevant background knowledge also led to a gain in AUC.CONCLUSIONBRLp enables tunable structure priors to be incorporated during Bayesian classification rule learning that integrates data and knowledge as demonstrated using lung cancer biomarker data.

Refinement and selection heuristics in subgroup discovery and classification rule learning

Classification rules and rules describing interesting subgroups are important components of descriptive machine learning. Rule learning algorithms typically proceed in two phases: rule refinement selects conditions for specializing the rule, and rule selection selects the final rule among several rule candidates. While most conventional algorithms use the same heuristic for guiding both phases, recent research indicates that the use of two separate heuristics is conceptually better justified, improves the coverage of positive examples, and may result in better classification accuracy. The paper presents and evaluates two new beam search rule learning algorithms: DoubleBeam-SD for subgroup discovery and DoubleBeam-RL for classification rule learning. The algorithms use two separate beams and can combine various heuristics for rule refinement and rule selection, which widens the search space and allows for finding rules with improved quality. In the classification rule learning setting, the experimental results confirm previously shown benefits of using two separate heuristics for rule refinement and rule selection. In subgroup discovery, DoubleBeam-SD algorithm variants outperform several state-of-the-art related algorithms.

Learning decision rules using adaptive technologies: a hybrid approach based on sequential covering

Abstract: Sequential covering strategies are commonly used in rule learning algorithms, as they apply separate and conquer approaches, in which the task of finding a complete rule base is reduced to a sequence of subproblems; each solution to a subproblem consists in adding a single rule. We propose an alternative for rule learning, namely the use of adaptive devices whose behavior is defined by a dynamic set of rules. In order to integrate features from rule learning methods and adaptive technologies, this paper presents a hybrid approach to learn classification rules, such that the set of rules is dynamically modified by adding or removing rules. The results have been promising towards applying adaptive technology to learn rules directly from data.

CoABCMiner: An Algorithm for Cooperative Rule Classification System Based on Artificial Bee Colony

In data mining, classification rule learning extracts the knowledge in the representation of IF_THEN rule which is comprehensive and readable. It is a challenging problem due to the complexity of data sets. Various meta-heuristic machine learning algorithms are proposed for rule learning. Cooperative rule learning is the discovery process of all classification rules with a single run concurrently. In this paper, a novel cooperative rule learning algorithm, called CoABCMiner, based on Artificial Bee Colony is introduced. The proposed algorithm handles the training data set and discovers the classification model containing the rule list. Token competition, new updating strategy used in onlooker and employed phases, and new scout bee mechanism are proposed in CoABCMiner to achieve cooperative learning of different rules belonging to different classes. We compared the results of CoABCMiner with several state-of-the-art algorithms using 14 benchmark data sets. Non parametric statistical tests, such as Friedman test, post hoc test, and contrast estimation based on medians are performed. Nonparametric tests determine the similarity of control algorithm among other algorithms on multiple problems. Sensitivity analysis of CoABCMiner is conducted. It is concluded that CoABCMiner can be used to discover classification rules for the data sets used in experiments, efficiently.

An ant colony-based semi-supervised approach for learning classification rules

Semi-supervised learning methods create models from a few labeled instances and a great number of unlabeled instances. They appear as a good option in scenarios where there is a lot of unlabeled data and the process of labeling instances is expensive, such as those where most Web applications stand. This paper proposes a semi-supervised self-training algorithm called Ant-Labeler. Self-training algorithms take advantage of supervised learning algorithms to iteratively learn a model from the labeled instances and then use this model to classify unlabeled instances. The instances that receive labels with high confidence are moved from the unlabeled to the labeled set, and this process is repeated until a stopping criteria is met, such as labeling all unlabeled instances. Ant-Labeler uses an ACO algorithm as the supervised learning method in the self-training procedure to generate interpretable rule-based models—used as an ensemble to ensure accurate predictions. The pheromone matrix is reused across different executions of the ACO algorithm to avoid rebuilding the models from scratch every time the labeled set is updated. Results showed that the proposed algorithm obtains better predictive accuracy than three state-of-the-art algorithms in roughly half of the datasets on which it was tested, and the smaller the number of labeled instances, the better the Ant-Labeler performance.

An improvement to the classification based on the measurement of the similarity quality using fuzzy relations

The learning of classification rules is a classic problem of the automatic learning. The algorithm IRBASIR for the induction of classification rules based on similaridad relations allows to discover knowledge starting from decision systems that contain features with continuous and discrete domains. This algorithm has shown to obtain higher results than other well-known algorithms. In this article, several modifications to this algorithm based on the Fuzzy sets theory are proposed, taking into account the measure quality of similarity. The experimental results show that using the fuzzy sets theory allow to obtain higher results than the original algorithm.

Does modeling lead to more accurate classification?: A study of relative efficiency in linear classification

Classification arises in a wide range of applications. A variety of statistical tools have been developed for learning classification rules from data. Understanding of their relative merits and comparisons help users to choose a proper method in practice. This paper focuses on theoretical comparison of model-based classification methods in statistics with algorithmic methods in machine learning in terms of the error rate. Extending Efron’s comparison of logistic regression with linear discriminant analysis (LDA) under the normal setting, we contrast such algorithmic methods as the support vector machine (SVM) and boosting with the LDA and logistic regression and study their relative efficiencies in reducing the error rate based on the limiting behavior of the classification boundary of each method. We show that algorithmic methods are generally less effective than model-based methods in the normal setting. In particular, loss of efficiency in error rate is typically about 33% to 60% for the SVM and 50% to 80% for boosting when compared to the LDA. However, a smooth variant of the SVM is shown to be even more efficient than logistic regression. In addition to the theoretical study, we present results from numerical experiments under various settings for comparisons of finite-sample performance and robustness to mislabeling and model misspecification.

An Application on Text Classification Based on Granular Computing

Machine learning is the key to text classification, a granular computing approach to machine learning is applied to learning classification rules by considering the two basic issues: concept formation and concept relationships identification. In this paper, we concentrate on the selection of a single granule in each step to construct a granule network. A classification rule induction method is proposed.

Speeding up multiple instance learning classification rules on GPUs

Multiple instance learning is a challenging task in supervised learning and data mining. However, algorithm performance becomes slow when learning from large-scale and high-dimensional data sets. Graphics processing units (GPUs) are being used for reducing computing time of algorithms. This paper presents an implementation of the G3P-MI algorithm on GPUs for solving multiple instance problems using classification rules. The GPU model proposed is distributable to multiple GPUs, seeking for its scalability across large-scale and high-dimensional data sets. The proposal is compared to the multi-threaded CPU algorithm with streaming SIMD extensions parallelism over a series of data sets. Experimental results report that the computation time can be significantly reduced and its scalability improved. Specifically, an speedup of up to 149 $$\times $$ can be achieved over the multi-threaded CPU algorithm when using four GPUs, and the rules interpreter achieves great efficiency and runs over 108 billion genetic programming operations per second.

SUBGROUP DISCOVERY OF THE MODY GENES;

Background: The pandemic of metabolic disorders is accelerating in the urbanized world posing huge burden to healthand economy. The key pioneer to most of the metabolic disorders is Diabetes Mellitus. A newly discovered form of diabetes is MaturityOnset Diabetes of the Young (MODY). MODY is a monogenic form of diabetes. It is inherited as autosomal dominant disorder. Till to date11 different MODY genes have been reported. Objective: This study aims to discover subgroups from the biological text documentsrelated to these genes in public domain database. Data Source: The data set was obtained from PubMed. Period: September-December,2011. Materials and Methodology: APRIORI-SD subgroup discovery algorithm is used for the task of discovering subgroups. A wellknown association rule learning algorithm APRIORI is first modified into classification rule learning algorithm APRIORI-C. APRIORI-Calgorithm generates the rule from the discretized dataset with the minimum support set to 0.42% with no confidence threshold. Total 580rules are generated at the given support. APRIOIR-C is further modified by making adaptation into APRIORI-SD. Results: Experimentalresults demonstrate that APRIORI discovers the substantially smaller rule sets; each rule has higher support and significance. The rulesthat are obtained by APRIORI-C are ordered by weighted relative accuracy. Conclusion: Only first 66 rules are ordered as they cover therelation between all the 11 MODY genes with each other. These 66 rules are further organized into 11 different subgroups. The evaluationof obtained results from literature shows that APRIORI-SD is a competitive subgroup discovery algorithm. All the association amonggenes proved to be true.

A novel algorithm for fuzzy rule induction in data mining

Rule induction as a method of constructing classifiers is of particular interest to data mining because it generates models in the form of If-Then rules which are more expressive and easier for humans to comprehend and check. Several induction algorithms have been developed to learn classification rules. However, most of these algorithms are based on ‘crisp’ data and produce ‘crisp’ models. This paper presents FuzzySRI, a novel algorithm based on the techniques of fuzzy sets and fuzzy logic for inducing fuzzy classification rules. The algorithm possesses the clear knowledge representation capability of rule induction methods and the ability of fuzzy techniques to handle vague information. Experimental results show that FuzzySRI can outperform other fuzzy and non-fuzzy learning systems in terms of predictive accuracy, comprehensibility, and computational efficiency. It is also shown that FuzzySRI can be successfully applied to an industrial application concerning the automatic identification of machine faults.

Semi-supervised learning on closed set lattices

We propose a new approach for semi-supervised learning using closed set lattices, which have been recently used for frequent pattern mining within the framework of the data analysis technique of Formal Concept Analysis FCA. We present a learning algorithm, called SELF SEmi-supervised Learning via FCA, which performs as a multiclass classifier and a label ranker for mixed-type data containing both discrete and continuous variables, while only few learning algorithms such as the decision tree-based classifier can directly handle mixed-type data. From both labeled and unlabeled data, SELF constructs a closed set lattice, which is a partially ordered set of data clusters with respect to subset inclusion, via FCA together with discretizing continuous variables, followed by learning classification rules through finding maximal clusters on the lattice. Moreover, it can weight each classification rule using the lattice, which gives a partial order of preference over class labels. We illustrate experimentally the competitive performance of SELF in classification and ranking compared to other learning algorithms using UCI datasets.

ON DESIGNING FUZZY RULE-BASED MULTICLASSIFICATION SYSTEMS BY COMBINING FURIA WITH BAGGING AND FEATURE SELECTION

In this work, we conduct a study considering a fuzzy rule-based multiclassification system design framework based on Fuzzy Unordered Rule Induction Algorithm (FURIA). This advanced method serves as the fuzzy classification rule learning algorithm to derive the component classifiers considering bagging and feature selection. We develop an exhaustive study on the potential of bagging and feature selection to design a final FURIA-based fuzzy multiclassifier dealing with high dimensional data. Several parameter settings for the global approach are tested when applied to twenty one popular UCI datasets. The results obtained show that FURIA-based fuzzy multiclassifiers outperform the single FURIA classifier and are competitive with C4.5 multiclassifiers and random forests.