Recursive-Rule Extraction Research Articles

Why Do Tree Ensemble Approximators Not Outperform the Recursive-Rule eXtraction Algorithm?

Although machine learning models are widely used in critical domains, their complexity and poor interpretability remain problematic. Decision trees (DTs) and rule-based models are known for their interpretability, and numerous studies have investigated techniques for approximating tree ensembles using DTs or rule sets, even though these approximators often overlook interpretability. These methods generate three types of rule sets: DT based, unordered, and decision list based. However, very few metrics exist that can distinguish and compare these rule sets. Therefore, the present study proposes an interpretability metric to allow for comparisons of interpretability between different rule sets and investigates the interpretability of the rules generated by the tree ensemble approximators. We compare these rule sets with the Recursive-Rule eXtraction algorithm (Re-RX) with J48graft to offer insights into the interpretability gap. The results indicate that Re-RX with J48graft can handle categorical and numerical attributes separately, has simple rules, and achieves a high interpretability, even when the number of rules is large. RuleCOSI+, a state-of-the-art method, showed significantly lower results regarding interpretability, but had the smallest number of rules.

One-Dimensional Convolutional Neural Networks with Feature Selection for Highly Concise Rule Extraction from Credit Scoring Datasets with Heterogeneous Attributes

Convolution neural networks (CNNs) have proven effectiveness, but they are not applicable to all datasets, such as those with heterogeneous attributes, which are often used in the finance and banking industries. Such datasets are difficult to classify, and to date, existing high-accuracy classifiers and rule-extraction methods have not been able to achieve sufficiently high classification accuracies or concise classification rules. This study aims to provide a new approach for achieving transparency and conciseness in credit scoring datasets with heterogeneous attributes by using a one-dimensional (1D) fully-connected layer first CNN combined with the Recursive-Rule Extraction (Re-RX) algorithm with a J48graft decision tree (hereafter 1D FCLF-CNN). Based on a comparison between the proposed 1D FCLF-CNN and existing rule extraction methods, our architecture enabled the extraction of the most concise rules (6.2) and achieved the best accuracy (73.10%), i.e., the highest interpretability–priority rule extraction. These results suggest that the 1D FCLF-CNN with Re-RX with J48graft is very effective for extracting highly concise rules for heterogeneous credit scoring datasets. Although it does not completely overcome the accuracy–interpretability dilemma for deep learning, it does appear to resolve this issue for credit scoring datasets with heterogeneous attributes, and thus, could lead to a new era in the financial industry.

Detection of Lower Albuminuria Levels and Early Development of Diabetic Kidney Disease Using an Artificial Intelligence-Based Rule Extraction Approach

The aim of the present study was to determine the lowest cut-off value for albuminuria levels, which can be used to detect diabetic kidney disease (DKD) using the urinary albumin-to-creatinine ratio (UACR). National Health and Nutrition Examination Survey (NHANES) data for US adults were used, and participants were classified as having diabetes or prediabetes based on a self-report and physiological measures. The study dataset comprised 942 diabetes and 524 prediabetes samples. This study clarified the significance of the lower albuminuria (UACR) levels, which can detect DKD, using an artificial intelligence-based rule extraction approach. The diagnostic rules (15 concrete rules) for both samples were extracted using a recursive-rule eXtraction (Re-RX) algorithm with continuous attributes (continuous Re-RX) to discriminate between prediabetes and diabetes datasets. Continuous Re-RX showed high test accuracy (77.56%) and a large area under the receiver operating characteristics curve (75%), which derived the two cut-off values (6.1 mg/g Cr and 71.00 mg/g Cr) for the lower albuminuria level in the UACR to detect early development of DKD. The early cut-off values for normoalbuminuria (NA) and microalbuminuria (MA) will be determined to help detect CKD and DKD, and to detect diabetes before MA develop and to prevent diabetic complications.

A rule extraction approach to explore the upper limit of hemoglobin during anemia treatment in patients with predialysis chronic kidney disease

Abstract Background In patients with predialysis chronic kidney disease (CKD) and anemia, maintaining appropriate hemoglobin (Hb) levels during treatment for renal anemia is critical. The risk of cardiovascular events increases when target Hb levels are ≥13 g/dL. However, few studies have extensively assessed the upper limit of Hb levels in renal anemia clinics. Therefore, considering the importance of variations in Hb levels, the aim of the present study was to explore the upper limit of Hb levels during anemia treatment in predialysis CKD patients utilizing rule extraction. Methods Data were obtained from the University of California–Irvine Machine Learning Repository. Patients with >2.0 mg/dL of serum creatinine were excluded from the analysis. The remaining 243 patients were classified into either a predialysis CKD group (n = 100) or a non-predialysis CKD (stages 1 and 2) group (n = 143). We explored the upper limit of Hb levels during anemia treatment in predialysis CKD patients utilizing rule extraction. The Recursive-Rule eXtraction (Re-RX) algorithm with J48graft was used to extract the classification rules. Results Three concrete and concise derived rules were used for anemia treatment. Re-RX with J48graft showed very high classification accuracy (95.18%) for the test dataset and a larger area under the receiver operating characteristic curve (94.4%) using 10 runs of 5-fold cross-validation. We also found an upper-limit Hb level of 12.9 g/dL during anemia treatment utilizing rule extraction to help avoid the risk of cardiovascular events. Conclusions To the best of our knowledge, this is the first study using rule extraction to explore the upper-limit Hb level to maintain during renal anemia treatment.

Non-invasive prediction of non-alcoholic steatohepatitis in Japanese patients with morbid obesity by artificial intelligence using rule extraction technology.

AIMTo construct a non-invasive prediction algorithm for predicting non-alcoholic steatohepatitis (NASH), we investigated Japanese morbidly obese patients using artificial intelligence with rule extraction technology.METHODSConsecutive patients who required bariatric surgery underwent a liver biopsy during the operation. Standard clinical, anthropometric, biochemical measurements were used as parameters to predict NASH and were analyzed using rule extraction technology. One hundred and two patients, including 79 NASH and 23 non-NASH patients were analyzed in order to create the prediction model, another cohort with 77 patients including 65 NASH and 12 non-NASH patients were analyzed to validate the algorithm.RESULTSAlanine aminotransferase, C-reactive protein, homeostasis model assessment insulin resistance, albumin were extracted as predictors of NASH using a recursive-rule extraction algorithm. When we adopted the extracted rules for the validation cohort using a highly accurate rule extraction algorithm, the predictive accuracy was 79.2%. The positive predictive value, negative predictive value, sensitivity and specificity were 88.9%, 35.7%, 86.2% and 41.7%, respectively.CONCLUSIONWe successfully generated a useful model for predicting NASH in Japanese morbidly obese patients based on their biochemical profile using a rule extraction algorithm.

Use of a Deep Belief Network for Small High-Level Abstraction Data Sets Using Artificial Intelligence with Rule Extraction.

We describe a simple method to transfer from weights in deep neural networks (NNs) trained by a deep belief network (DBN) to weights in a backpropagation NN (BPNN) in the recursive-rule eXtraction (Re-RX) algorithm with J48graft (Re-RX with J48graft) and propose a new method to extract accurate and interpretable classification rules for rating category data sets. We apply this method to the Wisconsin Breast Cancer Data Set (WBCD), the Mammographic Mass Data Set, and the Dermatology Dataset, which are small, high-abstraction data sets with prior knowledge. After training these three data sets, our proposed rule extraction method was able to extract accurate and concise rules for deep NNs trained by a DBN. These results suggest that our proposed method could help fill the gap between the very high learning capability of DBNs and the very high interpretability of rule extraction algorithms such as Re-RX with J48graft.

High Accuracy-priority Rule Extraction for Reconciling Accuracy and Interpretability in Credit Scoring

Accuracy and interpretability are two perspectives that are difficult to balance; this is referred to as the accuracy-interpretability dilemma. If credit models gain interpretability, they lose accuracy, and vice versa. Researchers continue to develop an array of very complicated predictive models; however, the finance industry needs interpretable models that can be used in actual practice. Especially, advanced sequential ensembles are seldom considered in credit scoring. Therefore, it is worthwhile to explore new rule extraction methods capable of building sequential ensemble classifiers that are effective for credit scoring. To enhance the accuracy and interpretability of extracted rules, we extend continuous recursive-rule extraction (continuous Re-RX) to a high accuracy-priority rule extraction method referred to as continuous Re-RX with J48graft. Continuous Re-RX with J48graft uses a recursive approach called subdivision. This approach consists of a backpropagation neural network, pruning, and a J48graft decision tree for mixed datasets (those containing discrete and continuous attributes) to construct a high accuracy-priority rule extraction method. Compared with previous rule extraction methods for Australian- and German-based datasets, continuous Re-RX with J48graft achieved the highest accuracies, 88.4 and 79.0%, respectively, using tenfold cross validation (CV) and the Friedman and Bonferroni–Dunn tests, and 87.82 and 78.4%, respectively, using 10 runs of tenfold CV, with the best Friedman score. We also demonstrate how continuous Re-RX with J48graft overcomes the accuracy-interpretability dilemma based on its performance. We believe that continuous Re-RX with J48graft can help overcome the accuracy-interpretability dilemma for transparency of Big Data in financial situations and for industrial applications.

Recursive Rule Extraction from NN using Reverse Engineering Technique

This paper proposes an algorithm named Reverse Engineering Recursive Rule Extraction (RE-Re-RX) for symbolic rule extraction from neural network with mixed attributes. RE-Re-RX algorithm is an extension of the existing Recursive Rule Extraction (Re-RX) algorithm. Re-RX algorithm generates disjoint rules for continuous and discrete attributes. The algorithm first generates rules for discrete attributes. A rule for discrete attributes is further refined recursively if it does not produce satisfactory result. A rule is refined by generating rules with the discrete attributes (if present) that are not covered by the rule or else the process is terminated by generating rules with the continuous attributes (if present). The novelty of the proposed RE-Re-RX algorithm lies in generating rules for continuous attributes. Re-RX generates linear hyper plane for continuous attributes which may not be able to deal with the non-linearity present in data. To overcome this limitation RE-Re-RX algorithm generates simple rules for continuous attributes in the form of input data ranges and target. RE-Re-RX uses the concept of Rule Extraction by Reverse Engineering the NN (RxREN) algorithm in slightly different way to generate rules. RxREN only uses misclassified patterns, whereas RE-Re-RX uses both classified and misclassified patterns of each continuous attribute to calculate input data ranges for constructing rules. The proposed algorithm is validated with six benchmark datasets. The experimental results clearly show the superiority of the proposed algorithm to Re-RX.

Synergy effects between grafting and subdivision in Re-RX with J48graft for the diagnosis of thyroid disease

Numerous methods for diagnosing thyroid disease have been developed, but the majority of these are black-box models. By contrast, the Recursive-Rule Extraction algorithm with J48graft is a white-box model that can provide highly accurate and concise classification rules. However, the potential capabilities of Re-RX with J48graft in terms of rule extraction remain unknown. Therefore, the aim of the present study was to elucidate the synergy effects between grafting and subdivision in Re-RX with J48graft, which work effectively in combination to extract highly accurate and concise classification rules for the diagnosis of thyroid disease. In the present study, I demonstrate how grafting and subdivision can extract highly accurate and concise classification rules from the Thyroid dataset, which is a large and highly imbalanced dataset consisting of 7200 medical records classified as normally functioning thyroid, hypothyroidism, or hyperthyroidism. I also provide the theoretical explanation underlying the excellent synergy effects between the two processes. Re-RX with J48graft not only achieved the most accurate classification rules, but also extracted simple and concrete concise classification rules for majority class samples. In addition, compared with previous methods, Re-RX with J48graft extracted rules with fewer antecedents. The maximum accuracy of the extracted rules was very high, at 97.02%. These findings suggest that Re-RX with J48graft can extract highly accurate and concise rules, which could assist healthcare professionals in the diagnosis of thyroid disease and help improve the level of care.

Recursive-Rule Extraction Algorithm With J48graft And Applications To Generating Credit Scores

Abstract The purpose of this study was to generate more concise rule extraction from the Recursive-Rule Extraction (Re-RX) algorithm by replacing the C4.5 program currently employed in Re-RX with the J48graft algorithm. Experiments were subsequently conducted to determine rules for six different two-class mixed datasets having discrete and continuous attributes and to compare the resulting accuracy, comprehensibility and conciseness. When working with the CARD1, CARD2, CARD3, German, Bene1 and Bene2 datasets, Re-RX with J48graft provided more concise rules than the original Re-RX algorithm. The use of Re-RX with J48graft resulted in 43.2%, 37% and 21% reductions in rules in the case of the German, Bene1 and Bene2 datasets compared to Re-RX. Furthermore, the Re-RX with J48graft showed 8.87% better accuracy than the Re-RX algorithm for the German dataset. These results confirm that the application of Re-RX in conjunction with J48graft has the capacity to facilitate migration from existing data systems toward new concise analytic systems and Big Data.

Rule extraction using Recursive-Rule extraction algorithm with J48graft combined with sampling selection techniques for the diagnosis of type 2 diabetes mellitus in the Pima Indian dataset

Abstract Diabetes is a complex disease that is increasing in prevalence around the world. Type 2 diabetes mellitus (T2DM) accounts for about 90–95% of all diagnosed adult cases of diabetes. Most present diagnostic methods for T2DM are black-box models, which are unable to provide the reasons underlying diagnosis to physicians; therefore, algorithms that can provide further insight are needed. Rule extraction can provide such explanations; however, in the medical setting, extracted rules must be not only highly accurate, but also simple and easy to understand. The Recursive-Rule eXtraction (Re-RX) algorithm is a “white-box” model that provides highly accurate classification. However, due to its recursive nature, it tends to generate more rules than other algorithms. Therefore, in this study, we propose the use of a rule extraction algorithm, Re-RX with J48graft, combined with sampling selection techniques (sampling Re-RX with J48graft) to achieve highly accurate, concise, and interpretable classification rules for the Pima Indian Diabetes (PID) dataset, which comprises 768 samples with two classes (diabetes or non-diabetes) and eight continuous attributes. The use of this algorithm resulted in an average accuracy of 83.83% after 10 runs of 10-fold cross validation. Sampling Re-RX with J48 graft achieved substantially better accuracy and provided a considerably fewer average number of rules and antecedents than the original Re-RX algorithm. These results suggest that sampling Re-RX with J48graft provides more accurate, concise, and interpretable extracted rules than previous algorithms, and is therefore more suitable for medical decision making, including the diagnosis of T2DM.

Accuracy of rule extraction using a recursive-rule extraction algorithm with continuous attributes combined with a sampling selection technique for the diagnosis of liver disease

Abstract Although liver cancer is the second most common cause of death from cancer worldwide, because of the limited accuracy and interpretability of extracted classification rules, the diagnosis of liver disease remains difficult. In addition, hepatitis, which is inflammation of the liver, can progress to fibrosis, cirrhosis, or even liver cancer. Numerous methods for diagnosing liver disease have been applied, but most current diagnostic methods are black box models that cannot adequately reveal information hidden in the data. In the medical setting, extracted rules must be not only highly accurate, but also highly interpretable. The Recursive-Rule eXtraction (Re-RX) algorithm is a white box model that generates highly accurate and interpretable classification rules on the basis of both discrete and continuous attributes; however, it tends to generate more rules than other rule extraction algorithms. The objectives of this study were to use a new rule extraction algorithm, Continuous Re-RX combined with sampling selection techniques (Sampling-Continuous Re-RX), to achieve highly accurate and interpretable diagnostic rules for the BUPA and Hepatitis datasets and to quantify the associations between the presence and severity of ascites and serum biomarkers with the risk of developing hepatitis in consideration of Child-Pugh scores. The performance of Sampling-Continuous Re-RX was compared with existing techniques, and as a result, it was found to extract more accurate, concise, and interpretable rules for the BUPA and Hepatitis datasets compared with previous extraction algorithms. In addition, the rules extracted using the proposed method were close to the trade-off curve, which indicated that they were more accurate and interpretable, and therefore more suitable in the medical setting.

Survey on the Family of the Recursive-Rule Extraction Algorithm

Survey on the Family of the Recursive-Rule Extraction Algorithm

Use of a Recursive-Rule eXtraction algorithm with J48graft to achieve highly accurate and concise rule extraction from a large breast cancer dataset

Abstract To assist physicians in the diagnosis of breast cancer and thereby improve survival, a highly accurate computer-aided diagnostic system is necessary. Although various machine learning and data mining approaches have been devised to increase diagnostic accuracy, most current methods are inadequate. The recently developed Recursive-Rule eXtraction (Re-RX) algorithm provides a hierarchical, recursive consideration of discrete variables prior to analysis of continuous data, and can generate classification rules that have been trained on the basis of both discrete and continuous attributes. The objective of this study was to extract highly accurate, concise, and interpretable classification rules for diagnosis using the Re-RX algorithm with J48graft, a class for generating a grafted C4.5 decision tree. We used the Wisconsin Breast Cancer Dataset (WBCD). Nine research groups provided 10 kinds of highly accurate concrete classification rules for the WBCD. We compared the accuracy and characteristics of the rule set for the WBCD generated using the Re-RX algorithm with J48graft with five rule sets obtained using 10-fold cross validation (CV). We trained the WBCD using the Re-RX algorithm with J48graft and the average classification accuracies of 10 runs of 10-fold CV for the training and test datasets, the number of extracted rules, and the average number of antecedents for the WBCD. Compared with other rule extraction algorithms, the Re-RX algorithm with J48graft resulted in a lower average number of rules for diagnosing breast cancer, which is a substantial advantage. It also provided the lowest average number of antecedents per rule. These features are expected to greatly aid physicians in making accurate and concise diagnoses for patients with breast cancer.

Use of the recursive-rule extraction algorithm with continuous attributes to improve diagnostic accuracy in thyroid disease

Abstract Thyroid diseases, which often lead to thyroid dysfunction involving either hypo- or hyperthyroidism, affect hundreds of millions of people worldwide, many of whom remain undiagnosed; however, diagnosis is difficult because symptoms are similar to those seen in a number of other conditions. The objective of this study was to assess the effectiveness of the Recursive-Rule Extraction (Re-RX) algorithm with continuous attributes (Continuous Re-RX) in extracting highly accurate, concise, and interpretable classification rules for the diagnosis of thyroid disease. We used the 7200-sample Thyroid dataset from the University of California Irvine Machine Learning Repository, a large and highly imbalanced dataset that comprises both discrete and continuous attributes. We trained the dataset using Continuous Re-RX, and after obtaining the maximum training and test accuracies, the number of extracted rules, and the average number of antecedents, we compared the results with those of other extraction methods. Our results suggested that Continuous Re-RX not only achieved the highest accuracy for diagnosing thyroid disease compared with the other methods, but also provided simple, concise, and interpretable rules. Based on these results, we believe that the use of Continuous Re-RX in machine learning may assist healthcare professionals in the diagnosis of thyroid disease.