Compact Rule Sets Research Articles

NPFAM: Non-Proliferation Fuzzy ARTMAP for Image Classification in Content Based Image Retrieval

A Content-based Image Retrieval (CBIR) system employs visual features rather than manual annotation of images. The selection of optimal features used in classification of images plays a key role in its performance. Category proliferation problem has a huge impact on performance of systems using Fuzzy Artmap (FAM) classifier. The proposed CBIR system uses a modified version of FAM called Non-Proliferation Fuzzy Artmap (NPFAM). This is developed by introducing significant changes in the learning process and the modified algorithm is evaluated by extensive experiments. Results have proved that NPFAM classifier generates a more compact rule set and performs better than FAM classifier. Accordingly, the CBIR system with NPFAM classifier yields good retrieval.

Time to make it shorter: Plain English in our context

Plain English in medical and scientific writing is not one-size-fits-all, because audiences differ. Advice on writing plain English abounds. In 1946, George Orwell, best known as the author of 1984, formulated a much quoted, compact set of rules for clear writing. The present article explores the relevance of his rules to medical writing, makes recommendations how to apply them, and adds further rules specific to our field of writing.

A novel business analytics approach and case study – fuzzy associative classifier based on information gain and rule-covering

Associative classification has attracted remarkable research attention for business analytics in recent years due to its merits in accuracy and understandability. It is deemed meaningful to construct an associative classifier with a compact set of rules (i.e., compactness), which is easy to understand and use in decision making. This paper presents a novel approach to fuzzy associative classification (namely Gain-based Fuzzy Rule-Covering classification, GFRC), which is a fuzzy extension of an effective classifier GARC. In GFRC, two desirable strategies are introduced to enhance the compactness with accuracy. One strategy is fuzzy partitioning for data discretization to cope with the ‘sharp boundary problem’, in that simulated annealing is incorporated based on the information entropy measure; the other strategy is a data-redundancy resolution coupled with the rule-covering treatment. Data experiments show that GFRC had good accuracy, and was significantly advantageous over other classifiers in compactness. Moreover, GFRC is applied to a real-world case for predicting the growth of sellers in an electronic marketplace, illustrating the classification effectiveness with linguistic rules in business decision support.

Very fast decision rules for classification in data streams

Data stream mining is the process of extracting knowledge structures from continuous, rapid data records. Many decision tasks can be formulated as stream mining problems and therefore many new algorithms for data streams are being proposed. Decision rules are one of the most interpretable and flexible models for predictive data mining. Nevertheless, few algorithms have been proposed in the literature to learn rule models for time-changing and high-speed flows of data. In this paper we present the very fast decision rules (VFDR) algorithm and discuss interesting extensions to the base version. All the proposed versions are one-pass and any-time algorithms. They work on-line and learn ordered or unordered rule sets. Algorithms designed to work with data streams should be able to detect changes and quickly adapt the decision model. In order to manage these situations we also present the adaptive extension (AVFDR) to detect changes in the process generating data and adapt the decision model. Detecting local drifts takes advantage of the modularity of the rule sets. In AVFDR, each individual rule monitors the evolution of performance metrics to detect concept drift. AVFDR prunes rules whenever a drift is signaled. This explicit change detection mechanism provides useful information about the dynamics of the process generating data, faster adaptation to changes and generates more compact rule sets. The experimental evaluation demonstrates that algorithms achieve competitive results in comparison to alternative methods and the adaptive methods are able to learn fast and compact rule sets from evolving streams.

Knowledge extraction and rule set compaction in XCS for non-Markov multi-step problems

The application of the XCS family of classifier systems within multi-step problems has shown promise, but XCS classifier systems usually solve those problems with a large number of classifiers. If they can generate compact and readily interpretable solutions when used in multi-step problems, it will enhance their applicability and usefulness in a wide range of robot control and knowledge discovery tasks. In this paper, we describe some methods developed for XCS to compact the rule set and acquire explicit policy knowledge for multi-step problems, especially non-Markov problems. Experimental results show the ability to obtain a very compact rule set out of the final population of classifiers with little or no degradation of overall performance.

Evolving optimum populations with XCS classifier systems

The main goal of the research direction is to extract building blocks of knowledge from a problem domain. Once extracted successfully, these building blocks are to be used in learning more complex problems of the domain, in an effort to produce a scalable learning classifier system (LCS). However, whilst current LCS (and other evolutionary computation techniques) discover good rules, they also create sub-optimum rules. Therefore, it is difficult to separate good building blocks of information from others without extensive post-processing. In order to provide richness in the LCS alphabet, code fragments similar to tree expressions in genetic programming are adopted. The accuracy-based XCS concept is used as it aims to produce maximally general and accurate classifiers, albeit the rule base requires condensation (compaction) to remove spurious classifiers. Serendipitously, this work on scalability of LCS produces compact rule sets that can be easily converted to the optimum population. The main contribution of this work is the ability to clearly separate the optimum rules from others without the need for expensive post-processing for the first time in LCS. This paper identifies that consistency of action in rich alphabets guides LCS to optimum rule sets.

Using genetic algorithm based knowledge refinement model for dividend policy forecasting

Dividend policy is one of most important managerial decisions affecting the firm value. Although there are many studies regarding decision-making problems, such as credit policy decisions through bankruptcy prediction and credit scoring, there is no research, to our knowledge, about dividend prediction or dividend policy forecasting using machine learning approaches in spite of the significance of dividends. For dealing with the problems involved in literature, we suggest a knowledge refinement model that can refine the multiple rules extracted through rule-based algorithms from dividend data sets by utilizing genetic algorithm (GA). The new technique, called “GAKR (genetic algorithm knowledge refinement)”, aims to combine the advantages of both knowledge consolidation and GA. The main result of the cross-validation procedure is the average accuracy rate of prediction in the five sets over the five iterations. The experiments show that GAKR model always outperforms other models in the performance of dividend policy prediction; we can predict future dividend policy more correctly than any other models. The major advantages of GAKR model can be summarized as follows: (1) Classification process of GAKR can be very fast with a compact set of rules. In other words, fast training mechanism of GAKR is possible regardless of data set sizes. (2) Multiple rules extracted by GAKR development process are much simpler and easier to understand. Moreover, GAKR model can discriminate redundant rules and inconsistent rules.

Kriya - An end-to-end Hierarchical Phrase-based MT System

This paper describes Kriya – a new statistical machine translation (SMT) system that uses hierarchical phrases, which were first introduced in the Hiero machine translation system (Chiang, 2007). Kriya supports both a grammar extraction module for synchronous context-free grammars (SCFGs) and a CKY-based decoder. There are several re-implementations of Hiero in the machine translation community, but Kriya offers the following novel contributions: (a) Grammar extraction in Kriya supports extraction of the full set of Hiero-style SCFG rules but also supports the extraction of several types of compact rule sets which leads to faster decoding for different language pairs without compromising the BLEU scores. Kriya currently supports extraction of compact SCFGs such as grammars with one non-terminal and grammar pruning based on certain rule patterns, and (b) The Kriya decoder offers some unique improvements in the implementation of cube-pruning, such as increasing diversity in the target language n-best output and novel methods for language model (LM) integration. The Kriya decoder can take advantage of parallelization using a networked cluster. Kriya supports both KENLM and SRILM for language model queries. This paper also provides several experimental results which demonstrate that the translation quality of Kriya compares favourably to the Moses (Koehn et al., 2007) phrase-based system in several language pairs while showing a substantial improvement for Chinese-English similar to Chiang (2007). We also quantify the model sizes for phrase-based and Hiero-style systems and also present experiments comparing variants of Hiero models.

Selecting informative rules with parallel genetic algorithm in classification problem

The classification system is very important for making decision and it has been attracted much attention of many researchers. Usually, the traditional classifiers are either domain specific or produce unsatisfactory results over classification problems with larger size and imbalanced data. Hence, genetic algorithms (GA) are recently being combined with traditional classifiers to find useful knowledge for making decision. Although, the main concerns of such GA-based system are the coverage of less search space and increase of computational cost with the growth of population. In this paper, a rule-based knowledge discovery model, combining C4.5 (a Decision Tree based rule inductive algorithm) and a new parallel genetic algorithm based on the idea of massive parallelism, is introduced. The prime goal of the model is to produce a compact set of informative rules from any kind of classification problem. More specifically, the proposed model receives a base method C4.5 to generate rules which are then refined by our proposed parallel GA. The strength of the developed system has been compared with pure C4.5 as well as the hybrid system (C4.5 + sequential genetic algorithm) on six real world benchmark data sets collected from UCI (University of California at Irvine) machine learning repository. Experiments on data sets validate the effectiveness of the new model. The presented results especially indicate that the model is powerful for volumetric data set.

Rough set analysis on call center metrics

Managers in call centers use metrics to measure organizational performance; unfortunately, these metrics only reveal how well service agents process calls. To achieve substantial quality improvement, managers need to obtain in-depth information from operational metrics such as first call resolution (FCR). This research incorporated rough set theory (RST) in analyzing FCR to produce decision rules. RST has been known to reduce the number of attributes and attribute values without affecting the original results. Data from a tariff (industry segment) call center in Taiwan were used in this study, and a four-step process was used to produce the final decision rules. The results were verified by a 10-fold cross-validation process against the decision rules produced without applying RST. The decision rules produced with RST are as effective as those produced without, but with reduced number of attributes and attribute values. This reduced decision rule set can help managers analyze the current operator procedures more efficiently and subsequently improve the call center efficiency. Without affecting effectiveness, the RST application reduces the needed number of attributes and attribute values to produce a more compact decision rule set, and this increased efficiency, in turn, allows managers to delve deeper into the operational factors.

HILK++: an interpretability-guided fuzzy modeling methodology for learning readable and comprehensible fuzzy rule-based classifiers

This work presents a methodology for building interpretable fuzzy systems for classification problems. We consider interpretability from two points of view: (1) readability of the system description and (2) comprehensibility of the system behavior explanations. The fuzzy modeling methodology named as Highly Interpretable Linguistic Knowledge (HILK) is upgraded. Firstly, a feature selection procedure based on crisp decision trees is carried out. Secondly, several strong fuzzy partitions are automatically generated from experimental data for all the selected inputs. For each input, all partitions are compared and the best one according to data distribution is selected. Thirdly, a set of linguistic rules are defined combining the previously generated linguistic variables. Then, a linguistic simplification procedure guided by a novel interpretability index is applied to get a more compact and general set of rules with a minimum loss of accuracy. Finally, partition tuning based on two efficient search strategies increases the system accuracy while preserving the high interpretability. Results obtained in several benchmark classification problems are encouraging because they show the ability of the new methodology for generating highly interpretable fuzzy rule-based classifiers while yielding accuracy comparable to that achieved by other methods like neural networks and C4.5. The best configuration of HILK will depend on each specific problem under consideration but it is important to remark that HILK is flexible enough (thanks to the combination of several algorithms in each modeling stage) to be easily adaptable to a wide range of problems.

GP-COACH: Genetic Programming-based learning of COmpact and ACcurate fuzzy rule-based classification systems for High-dimensional problems

In this paper we propose GP-COACH, a Genetic Programming-based method for the learning of COmpact and ACcurate fuzzy rule-based classification systems for High-dimensional problems. GP-COACH learns disjunctive normal form rules (generated by means of a context-free grammar) coded as one rule per tree. The population constitutes the rule base, so it is a genetic cooperative-competitive learning approach. GP-COACH uses a token competition mechanism to maintain the diversity of the population and this obliges the rules to compete and cooperate among themselves and allows the obtaining of a compact set of fuzzy rules. The results obtained have been validated by the use of non-parametric statistical tests, showing a good performance in terms of accuracy and interpretability.

Protein Superfamily Classification Using Fuzzy Rule-Based Classifier

In this paper, we have proposed a fuzzy rule-based classifier for assigning amino acid sequences into different superfamilies of proteins. While the most popular methods for protein classification rely on sequence alignment, our approach is alignment-free and so more human readable. It accounts for the distribution of contiguous patterns of n amino acids ( n-grams) in the sequences as features, alike other alignment-independent methods. Our approach, first extracts a plenty of features from a set of training sequences, then selects only some best of them, using a proposed feature ranking method. Thereafter, using these features, a novel steady-state genetic algorithm for extracting fuzzy classification rules from data is used to generate a compact set of interpretable fuzzy rules. The generated rules are simple and human understandable. So, the biologists can utilize them, for classification purposes, or incorporate their expertise to interpret or even modify them. To evaluate the performance of our fuzzy rule-based classifier, we have compared it with the conventional nonfuzzy C4.5 algorithm, beside some other fuzzy classifiers. This comparative study is conducted through classifying the protein sequences of five superfamily classes, downloaded from a public domain database. The obtained results show that the generated fuzzy rules are more interpretable, with acceptable improvement in the classification accuracy.

SGERD: A Steady-State Genetic Algorithm for Extracting Fuzzy Classification Rules From Data

This paper considers the automatic design of fuzzy-rule-based classification systems from labeled data. The performance of classifiers and the interpretability of generated rules are of major importance in these systems. In past research, some genetic-based algorithms have been used for the rule learning process. These genetic fuzzy systems have utilized different approaches to encode rules. In this paper, we have proposed a novel steady- state genetic algorithm to extract a compact set of good fuzzy rules from numerical data (SGERD). The selection mechanism of this algorithm is nonrandom, and only the best individuals can survive. Our approach is very simple and fast, and can be applied to high-dimensional problems with numerical attributes. To select the rules having high generalization capabilities, our algorithm makes use of some rule- and data-dependent parameters. We have also proposed an enhancing function that modifies the rule evaluation measures in order to assess the candidate rules more effectively before their selection. Experiments on some well-known data sets are performed to show the performance of SGERD.

Constructing accurate fuzzy classifiers: A new adaptive method for rule-weight specification

Different approaches to design fuzzy rule-based classification systems can be grouped into two main categories: descriptive and accurate. In the descriptive category, the emphasis is on the interpretability of the resulting classifier. The classifier is usually represented by a compact set of short fuzzy rules (i.e., with a few number of antecedent conditions) that make it a suitable tool for knowledge representation. In the accurate category, the generalization ability of the classifier is the main target in the design process and no attempt is made to use understandable fuzzy rules in constructing the rule base. In this paper, we propose a simple and efficient method to construct an accurate fuzzy classification system. We use rule-weight as a simple mechanism to tune the classifier and propose a new method of rule-weight specification for this purpose. Through computer simulations on some data sets from UCI repository, we show that the proposed scheme achieves better prediction accuracy compared with other fuzzy and non-fuzzy rule-based classification systems proposed in the past.

Pronunciation prediction with Default&Refine

The Default&Refine algorithm is a new rule-based learning algorithm that was developed as an accurate and efficient pronunciation prediction mechanism for speech processing systems. The algorithm exhibits a number of attractive properties including rapid generalisation from small training sets, good asymptotic accuracy, robustness to noise in the training data, and the production of compact rule sets. We describe the Default&Refine algorithm in detail and demonstrate its performance on two benchmarked pronunciation databases (the English OALD and Flemish FONILEX pronunciation dictionaries) as well as a newly-developed Afrikaans pronunciation dictionary. We find that the algorithm learns more efficiently (achieves higher accuracy on smaller data sets) than any of the alternative pronunciation prediction algorithms considered. In addition, we demonstrate the ability of the algorithm to generate an arbitrarily small rule set in such a way that the trade-off between rule set size and accuracy is well controlled. A conceptual comparison with alternative algorithms (including Dynamically Expanding Context, Transformation-Based Learning and Pronunciation by Analogy) clarifies the competitive performance obtained with Default&Refine.

Learning classifier system ensemble and compact rule set

This paper presents a learning classifier system ensemble for knowledge discovery from incremental data. The new ensemble was designed with a two-level architecture to improve the generalization ability. The new incoming cases are first bootstrapped to generate data as inputs to the first level classical learning classifier systems. The second level contains a plurality-vote module to determine the final classification by combining the classification results of the first level learning classifier systems. Each learning classifier system in the first level consists of two major modules, a genetic algorithm module for facilitating rule-discovery and a reinforcement learning module for adjusting the strength of the corresponding rules when rewards are received from the environment. We propose a revised Wilson's compact rule algorithm for generation of the compact rule set from the population set to improve the readability of the model. Two experiments were conducted. One was data mining of medical data and the other was steganalysis of images. The experimental results have shown that the new ensemble produced better performance on incremental data mining and better generalization than the single learning classifier system and other supervised learning methods. The results also showed that the compact rules were more interpretable.

A modified fuzzy min–max neural network with rule extraction and its application to fault detection and classification

The fuzzy min–max (FMM) network is a supervised neural network classifier that forms hyperboxes for classification and prediction. In this paper, we propose modifications to FMM in an attempt to improve its classification performance when a small number of large hyperboxes are formed in the network. Given a new input pattern, in addition to measuring the fuzzy membership function of the input pattern to the hyperboxes formed in FMM, an Euclidean distance measure is introduced for predicting the target class associated with the new input pattern. A rule extraction algorithm is also embedded into the modified FMM network. A confidence factor is calculated for each FMM hyperbox, and a user-defined threshold is used to prune the hyperboxes with low confidence factors. Fuzzy if– then rules are then extracted from the pruned network. The benefits of the proposed modifications are twofold, viz., to improve the performance of FMM when large hyperboxes are formed in the network; to facilitate the extraction of a compact rule set from FMM to justify its predictions. To assess the effectiveness of modified FMM, two benchmark pattern classification problems are experimented, and the results from different methods published in the literature are compared. In addition, a fault detection and classification problem with a set of real sensor measurements collected from a power generation plant is evaluated using modified FMM. The results obtained are analyzed and explained, and implications of the modified FMM network as a useful fault detection and classification tool in real environments are discussed.

Dynamic System Modeling Using Rules3 Induction Algorithm

This paper describes the use of a new induction algorithm to derive production rules for modelling dynamic systems. The algorithm, called RULES-3, is an efficient tool for extracting a compact set of IF-THEN rules from a collection of examples to classify objects into known categories. The paper summarises the operation of RULES-3 and presents the results obtained in the modelling of two linear systems and two non-linear systems.

Evolving accurate and compact classification rules with gene expression programming

Classification is one of the fundamental tasks of data mining. Most rule induction and decision tree algorithms perform a local, greedy search to generate classification rules that are often more complex than necessary. Evolutionary algorithms for pattern classification have recently received increased attention because they can perform global searches. In this paper, we propose a new approach for discovering classification rules by using gene expression programming (GEP), a new technique of genetic programming (GP) with linear representation. The antecedent of discovered rules may involve many different combinations of attributes. To guide the search process, we suggest a fitness function considering both the rule consistency gain and completeness. A multiclass classification problem is formulated as multiple two-class problems by using the one-against-all learning method. The covering strategy is applied to learn multiple rules if applicable for each class. Compact rule sets are subsequently evolved using a two-phase pruning method based on the minimum description length (MDL) principle and the integration theory. Our approach is also noise tolerant and able to deal with both numeric and nominal attributes. Experiments with several benchmark data sets have shown up to 20% improvement in validation accuracy, compared with C4.5 algorithms. Furthermore, the proposed GEP approach is more efficient and tends to generate shorter solutions compared with canonical tree-based GP classifiers.