Rule Extraction Techniques Research Articles

A Novel Integration of Data-Driven Rule Generation and Computational Argumentation for Enhanced Explainable AI

Explainable Artificial Intelligence (XAI) is a research area that clarifies AI decision-making processes to build user trust and promote responsible AI. Hence, a key scientific challenge in XAI is the development of methods that generate transparent and interpretable explanations while maintaining scalability and effectiveness in complex scenarios. Rule-based methods in XAI generate rules that can potentially explain AI inferences, yet they can also become convoluted in large scenarios, hindering their readability and scalability. Moreover, they often lack contrastive explanations, leaving users uncertain why specific predictions are preferred. To address this scientific problem, we explore the integration of computational argumentation—a sub-field of AI that models reasoning processes through defeasibility—into rule-based XAI systems. Computational argumentation enables arguments modelled from rules to be retracted based on new evidence. This makes it a promising approach to enhancing rule-based methods for creating more explainable AI systems. Nonetheless, research on their integration remains limited despite the appealing properties of rule-based systems and computational argumentation. Therefore, this study also addresses the applied challenge of implementing such an integration within practical AI tools. The study employs the Logic Learning Machine (LLM), a specific rule-extraction technique, and presents a modular design that integrates input rules into a structured argumentation framework using state-of-the-art computational argumentation methods. Experiments conducted on binary classification problems using various datasets from the UCI Machine Learning Repository demonstrate the effectiveness of this integration. The LLM technique excelled in producing a manageable number of if-then rules with a small number of premises while maintaining high inferential capacity for all datasets. In turn, argument-based models achieved comparable results to those derived directly from if-then rules, leveraging a concise set of rules and excelling in explainability. In summary, this paper introduces a novel approach for efficiently and automatically generating arguments and their interactions from data, addressing both scientific and applied challenges in advancing the application and deployment of argumentation systems in XAI.

Explainable based approach for the air quality classification on the granular computing rule extraction technique

Air pollution corresponds to one of the considerable challenges and disastrous sides of the environment that causes severe damage to all its biodiversity, including humans. As a result, establishing efficient, reliable, and interpretable methods and techniques to predict and control air quality is a must to preserve the environment and consider the necessary precautions. Most traditional machine learning models often lack transparency, making it challenging to interpret their decisions, especially in vital domains like air pollution. This paper proposes a novel approach that leverages granular computing to extract interpretable rules for air quality classification. We demonstrate the effectiveness of our approach through experiments on a real-world air quality dataset, showcasing the interpretability of the extracted rules and their accuracy in classifying air quality levels. The output of the proposed GrC model is a tree-like structure minimizing the entropy, allowing an easier interpretation of the classification results. A comparison is conducted with some widely used machine learning algorithms, including decision tree classifier, random forest, and CatBoost. The results indicate that the proposed granular computing rule extraction approach shows a competitive performance according to traditional black-box models in terms of accuracy (79%), transparency and reliability. The developed GrC model and the findings of this study not only contribute to advancing the field of air quality classification but also bear broader implications for environmental research and management for relevant and informed decision-making.

Two-Stage Intrusion Detection System in Intelligent Transportation Systems Using Rule Extraction Methods From Deep Neural Networks

In recent years, intrusion detection systems (IDSs) are offering effective solutions to protect various types of cyber-attacks in different networks such as Internet of Vehicles (IoVs) network in Intelligent Transportation Systems (ITS). Deep learning models have largely been leveraged by these intrusion detection systems to achieve better effectiveness results. However, deep learning models are black boxes, which limits their acceptability in decision systems. Also, they require powerful processing capabilities such as GPU, which limit their deployments in resource-constrained devices in IoV environment. To deal with these issues, we propose a two-stage IDS in ITS to discover suspicious network activity of In-Vehicles Networks (IVN) and vehicles to everything (V2X) networks. Our proposed IDS system uses rule extraction methods from deep learning models, i.e., deep neural networks in two stages. In the first stage, we analyze network traffic to distinguish between normal and attack traffic. If the traffic is found malicious, the second stage is invoked to identify the type of attack. To this end, we propose three variants of rule extraction. The first and the second variants are homogeneous, and they apply <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$DeepRed$</tex-math> </inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$HypInv$</tex-math> </inline-formula> rule extraction methods in both stages respectively. The third variant is heterogeneous, and it applies <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$HypInv$</tex-math> </inline-formula> in the first stage to perform binary classification, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$DeepRed$</tex-math> </inline-formula> in the second stage to perform attack classification. The key idea is to combine the advantages of rule extraction technique and two-stage IDS architecture to resource consumption and improve classification accuracy. The proposed IDS model was tested using four benchmark datasets, i.e. ISCXIDS2012, CIC-IDS2017, and CSE-CIC-IDS2018 datasets are used for external network communications and the car hacking dataset are used for in-vehicle communications. The evaluation results show that the homogeneous <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$DeepRed$</tex-math> </inline-formula> is the optimal one in all cases of IDS system with an accuracy scores ranging between 92.43%-98.32% under CIC-IDS2017 dataset, between 91.32%-99.46% under CSE-CIC-IDS2018 dataset, and between 96.05%-99.21% under Car-hacking dataset.

Rule extraction using ensemble of neural network ensembles

Neural networks are well-known for their impressive classification performance, and the ensemble learning technique acts as a catalyst to improve this performance even further by integrating multiple networks.However, neural network ensembles, like neural networks, are regarded as a black box because they cannot explain their decision-making process. As a result, despite their high classification performance, neural networks and their ensembles are unsuitable for some applications that require explainable decisions. However, the rule extraction technique can overcome this drawback by representing the knowledge learned by a neural network in the guise of interpretable decision rules. A rule extraction algorithm provides neural networks the ability to justify their classification responses using explainable classification rules. There are several rule extraction algorithms for extracting classification rules from neural networks, but only a few of them use neural network ensembles to generate rules. As a result, this paper proposes a rule extraction algorithm called Rule Extraction Using Ensemble of Neural Network Ensembles (RE-E-NNES) to demonstrate the high performance of neural network ensembles.RE-E-NNES extracts classification rules by ensembling several neural network ensembles. The results demonstrate the efficacy of the proposed RE-E-NNES algorithm in comparison to other existing rule extraction algorithms.

A Rule Extraction Technique Applied to Ensembles of Neural Networks, Random Forests, and Gradient-Boosted Trees

In machine learning, ensembles of models based on Multi-Layer Perceptrons (MLPs) or decision trees are considered successful models. However, explaining their responses is a complex problem that requires the creation of new methods of interpretation. A natural way to explain the classifications of the models is to transform them into propositional rules. In this work, we focus on random forests and gradient-boosted trees. Specifically, these models are converted into an ensemble of interpretable MLPs from which propositional rules are produced. The rule extraction method presented here allows one to precisely locate the discriminating hyperplanes that constitute the antecedents of the rules. In experiments based on eight classification problems, we compared our rule extraction technique to “Skope-Rules” and other state-of-the-art techniques. Experiments were performed with ten-fold cross-validation trials, with propositional rules that were also generated from ensembles of interpretable MLPs. By evaluating the characteristics of the extracted rules in terms of complexity, fidelity, and accuracy, the results obtained showed that our rule extraction technique is competitive. To the best of our knowledge, this is one of the few works showing a rule extraction technique that has been applied to both ensembles of decision trees and neural networks.

Rule extraction in unsupervised anomaly detection for model explainability: Application to OneClass SVM

OneClass SVM is a popular method for unsupervised anomaly detection. As many other methods, it suffers from the black box problem: it is difficult to justify, in an intuitive and simple manner, why the decision frontier is identifying data points as anomalous or non anomalous. This problem is being widely addressed for supervised models. However, it is still an uncharted area for unsupervised learning. In this paper, we evaluate several rule extraction techniques over OneClass SVM models, while presenting alternative designs for some of those algorithms. Furthermore, we propose algorithms for computing metrics related to eXplainable Artificial Intelligence (XAI) regarding the “comprehensibility”, “representativeness”, “stability” and “diversity” of the extracted rules. We evaluate our proposals with different data sets, including real-world data coming from industry. Consequently, our proposal contributes to extending XAI techniques to unsupervised machine learning models.

Can metafeatures help improve explanations of prediction models when using behavioral and textual data?

Machine learning models on behavioral and textual data can result in highly accurate prediction models, but are often very difficult to interpret. Rule-extraction techniques have been proposed to combine the desired predictive accuracy of complex "black-box" models with global explainability. However, rule-extraction in the context of high-dimensional, sparse data, where many features are relevant to the predictions, can be challenging, as replacing the black-box model by many rules leaves the user again with an incomprehensible explanation. To address this problem, we develop and test a rule-extraction methodology based on higher-level, less-sparse metafeatures. A key finding of our analysis is that metafeatures-based explanations are better at mimicking the behavior of the black-box prediction model, as measured by the fidelity of explanations.

A Two-Step Rule-Extraction Technique for a CNN

The explanation of the decisions provided by a model are crucial in a domain such as medical diagnosis. With the advent of deep learning, it is very important to explain why a classification is reached by a model. This work tackles the transparency problem of convolutional neural networks(CNNs). We propose to generate propositional rules from CNNs, because they are intuitive to the way humans reason. Our method considers that a CNN is the union of two subnetworks: a multi-layer erceptron (MLP) in the fully connected layers; and a subnetwork including several 2D convolutional layers and max-pooling layers. Rule extraction exhibits two main steps, with each step generating rules from each subnetwork of the CNN. In practice, we approximate the two subnetworks by two particular MLP models that makes it possible to generate propositional rules. We performed the experiments with two datasets involving images: MNISTdigit recognition; and skin-cancer diagnosis. With high fidelity, the extracted rules designated the location of discriminant pixels, as well as the conditions that had to be met to achieve the classification. We illustrated several examples of rules by their centroids and their discriminant pixels.

Field Level Security of the Sensitive Data in Large Datasets

The process of deriving useful and knowledgeable information from enormous quantity of data is Data Mining. During mining procedures, handling of the sensitive data has become important to protect data against illegal attacks and malicious access either during transmission or at rest. Association rule algorithm is one of the rule extraction techniques. The rules determined are either to be transferred over the public networks or to be rested for further use.The main objective of the Field Level Security of the Sensitive Data in Large Datasets is to extract the strong association rules from the large data sets and the outcomes are crafted to conceal the sensitive data. The datasets and the association rules involving the attributes with relationships and dependencies are modified through several approaches and to see that no sensitive association rule is derived from it[1]. Privacy preservation of the sensitive association rules in large datasets is to provide secrecy for the sensitive data. Presently, it has become quite important to safeguard the privacy of the users’ personal data from unauthorized persons. The usage of association rules in voluminous datasets has emerged to be advantageous to organizations [2]. In this paper, we present a novel approach which is applied for hiding sensitive association rules by utilizing the techniques of compression, encryption method ology on the original dataset, providing dataset with better immunity.

A Simple Convolutional Neural Network with Rule Extraction

Classification responses provided by Multi Layer Perceptrons (MLPs) can be explained by means of propositional rules. So far, many rule extraction techniques have been proposed for shallow MLPs, but not for Convolutional Neural Networks (CNNs). To fill this gap, this work presents a new rule extraction method applied to a typical CNN architecture used in Sentiment Analysis (SA). We focus on the textual data on which the CNN is trained with “tweets” of movie reviews. Its architecture includes an input layer representing words by “word embeddings”, a convolutional layer, a max-pooling layer, followed by a fully connected layer. Rule extraction is performed on the fully connected layer, with the help of the Discretized Interpretable Multi Layer Perceptron (DIMLP). This transparent MLP architecture allows us to generate symbolic rules, by precisely locating axis-parallel hyperplanes. Experiments based on cross-validation emphasize that our approach is more accurate than that based on SVMs and decision trees that substitute DIMLPs. Overall, rules reach high fidelity and the discriminative n-grams represented in the antecedents explain the classifications adequately. With several test examples we illustrate the n-grams represented in the activated rules. They present the particularity to contribute to the final classification with a certain intensity.

Symbolic interpretation of artificial neural networks based on multiobjective genetic algorithms and association rules mining

Rule extraction from neural networks is an important task. In practice, decision makers often settle for using less accurate, but comprehensible models, typically decision trees where the solutions are given in graphical form easily interpretable. The black-box rule extraction techniques, operating directly on the input-output relationship, are clearly superior to the restricted open-box methods, normally tailored for a specific architecture. This is especially important since most data miners today will use some kind of ensemble (instead of a single model) to maximize accuracy. Consequently, the ability to extract rules from any opaque model is a key demand for rule extraction techniques. This paper proposes a new multiobjective genetic method to extract knowledge from trained artificial neural network by using the association rules technique. The main aim of this hybridization is to extract the optimal rules from the neural network for further classification. The algorithm consists of two stages: the rule filtering phase which eliminates misleading rules by taking into account the support, the confidence and the lift measures, then, rule set optimization phase which finds the set of optimal rule sets by considering fidelity, coverage and complexity measures. The algorithm is evaluated on 05 UCI datasets. The experimental results show that the proposal provides interesting rules. Accuracy and comprehensibility are clearly improved, and subsequently, it can become a challengeable and trustful research field in the area of neural network rule extraction.

Derivation of simplified control rules from an optimal strategy for electric heating in a residential building

In France, 40% of buildings are heated with electrical devices causing high peak load in winter. In this context, advanced control systems could improve buildings energy management. More specifically, optimal strategies have been developed using a dynamic programming method in order to shift heating load, taking advantage of the building thermal mass. However, this optimization method is computationally intensive and can hardly be applied to real-time control. Statistical techniques can be used to derive near-optimal laws from the optimal control results. These rule extraction techniques model the relationship between explanatory variables and a response variable. This paper investigates the use of Beta regression model. This regression-based strategy was able to mimic the general characteristics of the optimization results with a small mean bias error (−6%) and greatly reduce computational effort (150 times faster). Given its simple mathematical formulation, it could be implemented in real-time building systems control.

Learning temporal rules to forecast instability in continuously monitored patients.

Inductive machine learning, and in particular extraction of association rules from data, has been successfully used in multiple application domains, such as market basket analysis, disease prognosis, fraud detection, and protein sequencing. The appeal of rule extraction techniques stems from their ability to handle intricate problems yet produce models based on rules that can be comprehended by humans, and are therefore more transparent. Human comprehension is a factor that may improve adoption and use of data-driven decision support systems clinically via face validity. In this work, we explore whether we can reliably and informatively forecast cardiorespiratory instability (CRI) in step-down unit (SDU) patients utilizing data from continuous monitoring of physiologic vital sign (VS) measurements. We use a temporal association rule extraction technique in conjunction with a rule fusion protocol to learn how to forecast CRI in continuously monitored patients. We detail our approach and present and discuss encouraging empirical results obtained using continuous multivariate VS data from the bedside monitors of 297 SDU patients spanning 29 346 hours (3.35 patient-years) of observation. We present example rules that have been learned from data to illustrate potential benefits of comprehensibility of the extracted models, and we analyze the empirical utility of each VS as a potential leading indicator of an impending CRI event.

Competitive interaction reasoning: A bio-inspired reasoning method for fuzzy rule based classification systems

In designing fuzzy rule based classification systems (FRBCSs), complex fuzzy rule extraction techniques and tuning membership functions are frequently used to enhance classification accuracy. However, these approaches not only decrease system's transparency which is the hallmark of fizzy design, but also are oftentimes computationally expensive and require multiple parameters to be optimized using a large amount of training data. In this paper, inspired by the so-called competitive behavior of mini-columns in the brain neuronal circuitry, we proposed a new reasoning method for fuzzy classifiers referred to as Competitive Interaction Reasoning (CIR) that employs the cumulative information provided by all fuzzy rules and adjusts the decision boundaries as if the membership functions are directly modified. This mechanism is mathematically implemented by a linear transformation and resembles the competitive interaction observed in brain neuronal columns. Cross-rule competition weights are optimized using Hebbian reinforcement learning. Using a large number of simulations on benchmark data sets, we show that the proposed CIR significantly improves classification accuracy without compromising interpretability of the fuzzy classifier. In addition, CIR can further facilitate formation of the fuzzy rules and incorporation of the expert knowledge by confining the destructive effects of noisy rules or expert inconsistencies. Experiments on 23 well-known benchmark data sets confirm high performance of CIR in comparison with a number of popular classifiers.

Modelling of an Extended Brutedl Algorithm for Rule Extraction

Complex predictive models obtain very high predictive performance; however, it is difficult to explain their complex mathematical design. Rule extraction techniques help to understand their designs by generating structures like decision list. BruteDL algorithm generates decision list from a dataset, and also addresses the overlapping rule problem of most decision list learners. However; it does not harness the power of complex predictive model. It also performs poorly with small dataset. Hence, this work aimed to create rule extraction technique by extending BruteDL and to address its poor performance with small dataset. A rule extraction technique named BruteDL-RET (Brute Decision List-Rule Extraction Technique) was modeled and implemented. A finite state automaton was used to model the technique. A functionality to generate supporting training set was included. Artificial Neural Networks (ANN) is chosen as the complex predictive model which serves as the oracle because it decides the class of each example. Decision list was generated using both the predictive model and the dataset it was trained with. The implementation was done using Java programming language. We prove that on the average BruteDL-RET is able to generate more accurate rules than BruteDL. We report on the performance of our model using dataset of UCI repository.

Application of a rule extraction algorithm family based on the Re-RX algorithm to financial credit risk assessment from a Pareto optimal perspective

Historically, the assessment of credit risk has proved to be both highly important and extremely difficult. Currently, financial institutions rely on the use of computer-generated credit scores for risk assessment. However, automated risk evaluations are currently imperfect, and the loss of vast amounts of capital could be prevented by improving the performance of computerized credit assessments. A number of approaches have been developed for the computation of credit scores over the last several decades, but these methods have been considered too complex without good interpretability and have therefore not been widely adopted. Therefore, in this study, we provide the first comprehensive comparison of results regarding the assessment of credit risk obtained using 10 runs of 10-fold cross validation of the Re-RX algorithm family, including the Re-RX algorithm, the Re-RX algorithm with both discrete and continuous attributes (Continuous Re-RX), the Re-RX algorithm with J48graft, the Re-RX algorithm with a trained neural network (Sampling Re-RX), NeuroLinear, NeuroLinear+GRG, and three unique rule extraction techniques involving support vector machines and Minerva from four real-life, two-class mixed credit-risk datasets. We also discuss the roles of various newly-extended types of the Re-RX algorithm and high performance classifiers from a Pareto optimal perspective. Our findings suggest that Continuous Re-RX, Re-RX with J48graft, and Sampling Re-RX comprise a powerful management tool that allows the creation of advanced, accurate, concise and interpretable decision support systems for credit risk evaluation. In addition, from a Pareto optimal perspective, the Re-RX algorithm family has superior features in relation to the comprehensibility of extracted rules and the potential for credit scoring with Big Data.

Active Learning-Based Pedagogical Rule Extraction.

Many of the state-of-the-art data mining techniques introduce nonlinearities in their models to cope with complex data relationships effectively. Although such techniques are consistently included among the top classification techniques in terms of predictive power, their lack of transparency renders them useless in any domain where comprehensibility is of importance. Rule-extraction algorithms remedy this by distilling comprehensible rule sets from complex models that explain how the classifications are made. This paper considers a new rule extraction technique, based on active learning. The technique generates artificial data points around training data with low confidence in the output score, after which these are labeled by the black-box model. The main novelty of the proposed method is that it uses a pedagogical approach without making any architectural assumptions of the underlying model. It can therefore be applied to any black-box technique. Furthermore, it can generate any rule format, depending on the chosen underlying rule induction technique. In a large-scale empirical study, we demonstrate the validity of our technique to extract trees and rules from artificial neural networks, support vector machines, and random forests, on 25 data sets of varying size and dimensionality. Our results show that not only do the generated rules explain the black-box models well (thereby facilitating the acceptance of such models), the proposed algorithm also performs significantly better than traditional rule induction techniques in terms of accuracy as well as fidelity.

Mining Building Energy Management System Data Using Fuzzy Anomaly Detection and Linguistic Descriptions

Building Energy Management Systems (BEMSs) are essential components of modern buildings that are responsible for minimizing energy consumption while maintaining occupant comfort. However, since indoor environment is dependent on many uncertain criteria, performance of BEMS can be suboptimal at times. Unfortunately, complexity of BEMSs, large amount of data, and interrelations between data can make identifying these suboptimal behaviors difficult. This paper proposes a novel Fuzzy Anomaly Detection and Linguistic Description (Fuzzy-ADLD)-based method for improving the understandability of BEMS behavior for improved state-awareness. The presented method is composed of two main parts: 1) detection of anomalous BEMS behavior; and 2) linguistic representation of BEMS behavior. The first part utilizes modified nearest neighbor clustering algorithm and fuzzy logic rule extraction technique to build a model of normal BEMS behavior. The second part of the presented method computes the most relevant linguistic description of the identified anomalies. The presented Fuzzy-ADLD method was applied to real-world BEMS system and compared against a traditional alarm-based BEMS. Six different scenarios were tested, and the presented Fuzzy-ADLD method identified anomalous behavior either as fast as or faster (an hour or more) than the alarm based BEMS. Furthermore, the Fuzzy-ADLD method identified cases that were missed by the alarm-based system, thus demonstrating potential for increased state-awareness of abnormal building behavior.

Aspects of Intelligent Systems Explanation

Intelligent systems encompass a wide range of software technologies including heuristic and normative expert systems, case-based reasoning systems, and neural networks. This field has been augmented in recent years by Web-based applications, such as recommender systems and the semantic Web. The uses of explanation facilities have their roots in heuristic rule-based expert systems and have long been touted as an important adjunct in intelligent decision support systems. However, in recent years, their uses have been explored in many other intelligent system technologies - particularly those making an impact in e-commerce such as recommender systems. This paper shows how explanation facilities work with a range of symbolic intelligent techniques and, when carefully designed, provide a range of benefits. The paper also shows how, despite being more difficult to augment with non-symbolic technologies, hybrid methods predominantly using rule-extraction techniques have provided moderate success for explanation facilities in a range of ad-hoc applications.

Rule extraction from support vector machines based on consistent region covering reduction

Due to good performance in classification and regression, support vector machines have attracted much attention and become one of the most popular learning machines in last decade. As a black box, the support vector machine is difficult for users’ understanding and explanation. In many application domains including medical diagnosis or credit scoring, understandability and interpretability are very important for the practicability of the learned models. To improve the comprehensibility of SVMs, we propose a rule extraction technique from support vector machines via analyzing the distribution of samples. We define the consistent region of samples in terms of classification boundary, and form a consistent region covering of the sample space. Then a covering reduction algorithm is developed for extracting compact representation of classes, thus a minimal set of decision rules is derived. Experiment analysis shows that the extracted models perform well in comparison with decision tree algorithms and other support vector machine rule extraction methods.