Neural Network Rule Research Articles

A note on knowledge discovery using neural networks and its application to credit card screening

We address an important issue in knowledge discovery using neural networks that has been left out in a recent article “Knowledge discovery using a neural network simultaneous optimization algorithm on a real world classification problem” by Sexton et al. [R.S. Sexton, S. McMurtrey, D.J. Cleavenger, Knowledge discovery using a neural network simultaneous optimization algorithm on a real world classification problem, European Journal of Operational Research 168 (2006) 1009–1018]. This important issue is the generation of comprehensible rule sets from trained neural networks. In this note, we present our neural network rule extraction algorithm that is very effective in discovering knowledge embedded in a neural network. This algorithm is particularly appropriate in applications where comprehensibility as well as accuracy are required. For the same data sets used by Sexton et al. our algorithm produces accurate rule sets that are concise and comprehensible, and hence helps validate the claim that neural networks could be viable alternatives to other data mining tools for knowledge discovery.

Development of a rough set-based fuzzy neural network for online monitoring of microdrilling

A rough set-based fuzzy neural network has been developed in this study for online monitoring of microdrilling. By applying the rough set theory, we obtained reduced rule sets from data samples as the fuzzy neural network rules. Then the neural network model was designed based on the reduced rule sets. The number of the network fuzzy rules was reduced. Shortcomings in high-dimensional fuzzy neural network, such as huge structure, were overcome. Data that were sampled in real-time from the spindle motor three-phase currents were processed by the trained network to monitor the microdrill wear online. The experiment results show that if the threshold is properly selected, the microdrill breakage will be effectively prevented by the monitoring.

Greedy rule generation from discrete data and its use in neural network rule extraction

This paper proposes a GRG (Greedy Rule Generation) algorithm, a new method for generating classification rules from a data set with discrete attributes. The algorithm is "greedy" in the sense that at every iteration, it searches for the best rule to generate. The criteria for the best rule include the number of samples and the size of subspaces that it covers, as well as the number of attributes in the rule. This method is employed for extracting rules from neural networks that have been trained and pruned for solving classification problems. The classification rules are extracted from the neural networks using the standard decompositional approach. Neural networks with one hidden layer are trained and the proposed GRG algorithm is applied to their discretized hidden unit activation values. Our experimental results show that neural network rule extraction with the GRG method produces rule sets that are accurate and concise. Application of GRG directly on three medical data sets with discrete attributes also demonstrates its effectiveness for rule generation.

Recursive Neural Network Rule Extraction for Data With Mixed Attributes

In this paper, we present a recursive algorithm for extracting classification rules from feedforward neural networks (NNs) that have been trained on data sets having both discrete and continuous attributes. The novelty of this algorithm lies in the conditions of the extracted rules: the rule conditions involving discrete attributes are disjoint from those involving continuous attributes. The algorithm starts by first generating rules with discrete attributes only to explain the classification process of the NN. If the accuracy of a rule with only discrete attributes is not satisfactory, the algorithm refines this rule by recursively generating more rules with discrete attributes not already present in the rule condition, or by generating a hyperplane involving only the continuous attributes. We show that for three real-life credit scoring data sets, the algorithm generates rules that are not only more accurate but also more comprehensible than those generated by other NN rule extraction methods.

Knowledge discovery in corporate events by neural network rule extraction

Over the last two decades, artificial neural networks (ANN) have been applied to solve a variety of problems such as pattern classification and function approximation. In many applications, it is desirable to extract knowledge from trained neural networks for the users to gain a better understanding of the network's solution. In this paper, we use a neural network rule extraction method to extract knowledge from 2222 dividend initiation and resumption events. We find that the positive relation between the short-term price reaction and the ratio of annualized dividend amount to stock price is primarily limited to 96 small firms with high dividend ratios. The results suggest that the degree of short-term stock price underreaction to dividend events may not be as dramatic as previously believed. The results also show that the relations between the stock price response and firm size is different across different types of firms. Thus, drawing the conclusions from the whole dividend event data may leave some important information unexamined. This study shows that neural network rule extraction method can reveal more knowledge from the data.

EXPLANATORY RULE EXTRACTION BASED ON THE TRAINED NEURAL NETWORK AND THE GENETIC PROGRAMMING

This paper deals with the use of neural network rule extraction techniques based on the Genetic Programming (GP) to build intelligent and explanatory evaluation systems. Recent development in algorithms that extract rules from trained neural networks enable us to generate classification rules in spite of their intrinsically black-box nature. However, in the original decompositional method looking at the internal structure of the networks, the comprehensive methods combining the output to the inputs using parameters are complicated. Then, in our paper, we utilized the GP to automatize the rule extraction process in the trained neural networks where the statements changed into a binary classification. Even though the production (classification) rule generation based on the GP alone are applicable straightforward to the underlying problems for decision making, but in the original GP method production rules include many statements described by arithmetic expressions as well as basic logical expressions, and it makes the rule generation process very complicated. Therefore, we utilize the neural network and binary classification to obtain simple and relevant classification rules in real applications by avoiding straightforward applications of the GP procedure to the arithmetic expressions. At first, the pruning process of weight among neurons is applied to obtain simple but substantial binary expressions which are used as statements is classification rules. Then, the GP is applied to generate ultimate rules. As applications, we generate rules to prediction of bankruptcy and creditworthiness for binary classifications, and the apply the method to multi-level classification of corporate bonds (rating) by using the financial indicators.

A hybrid model by clustering and evolving fuzzy rules for sales decision supports in printed circuit board industry

This research develops a hybrid model by integrating Self Organization Map (SOM) neural network, Genetic Algorithms (GA) and Fuzzy Rule Base (FRB) to forecast the future sales of a printed circuit board factory. This hybrid model encompasses two novel concepts: (1) clustering an FRB into different clusters, thus the interaction between fuzzy rules is reduced and a more accurate prediction model can be established, and (2) evolving an FRB by optimizing the number of fuzzy terms of the input and output variables, thus the prediction accuracy of the FRB is further improved. Numerical data of various affecting factors and actual demand of the past 5 years of the printed circuit board (PCB) factory are collected and inputted into the hybrid model for future monthly sales forecasting. Experimental results show the effectiveness of the hybrid model when comparing it with other approaches. However, the theoretical development of the validity of clustering an FRB into sub clusters remains to be proven.

Separating Core and Noncore Knowledge: An Application of Neural Network Rule Extraction to a Cross-National Study of Brand Image Perception

Recent advances in algorithms that extract rules from artificial neural networks make it feasible to use neural networks as a tool for acquiring knowledge hidden in the data. Findings are reported from the use of such algorithms to separate core and noncore knowledge in a cross-national study of automobile brand image perception. Respondents from five Western European countries have been asked to associate individual and corporate brand associations for a number of well-known automobile brands. Knowledge, expressed as concise and accurate rules that distinguish between the respondents' perceptions of German and Japanese brands, is extracted from trained neural networks. This paper explains how both core knowledge, which captures the perceptions shared by the respondents in all countries, and country-specific noncore knowledge can be acquired and differentiated by a proposed two-step approach to train and extract rules from a multi-neural network system. The experimental results show that, in addition to providing a better understanding of the differences and similarities in the brand image perceptions of consumers in various countries, the proposed approach also yields better predictive accuracy than a decision tree method.

Neural Network Rule Extraction by Using the Genetic Programming and Its Applications to Explanatory Classifications

This paper deals with the use of neural network rule extraction techniques based on the Genetic Programming (GP) to build intelligent and explanatory evaluation systems. Recent development in algorithms that extract rules from trained neural networks enable us to generate classification rules in spite of their intrinsically black-box nature. However, in the original decompositional method looking at the internal structure of the networks, the comprehensive methods combining the output to the inputs using parameters are complicated. Then, in our paper, we utilized the GP to automatize the rule extraction process in the trained neural networks where the statements changed into a binary classification. Even though the production (classification) rule generation based on the GP alone are applicable straightforward to the underlying problems for decision making, but in the original GP method production rules include many statements described by arithmetic expressions as well as basic logical expressions, and it makes the rule generation process very complicated. Therefore, we utilize the neural network and binary classification to obtain simple and relevant classification rules in real applications by avoiding straightforward applications of the GP procedure to the arithmetic expressions. At first, the pruning process of weight among neurons is applied to obtain simple but substantial binary expressions which are used as statements is classification rules. Then, the GP is applied to generate ultimate rules. As applications, we generate rules to prediction of bankruptcy and creditworthiness for binary classifications, and the apply the method to multi-level classification of corporate bonds (rating) by using the financial indicators.

A model of STDP based on spatially and temporally local information: Derivation and combination with gated decay

Temporal relationships between neuronal firing and plasticity have received significant attention in recent decades. Neurophysiological studies have shown the phenomenon of spike-timing-dependent plasticity (STDP). Various models were suggested to implement an STDP-like learning rule in artificial networks based on spiking neuronal representations. The rule presented here was developed under three constraints. First, it only depends on the information that is available at the synapse at the time of synaptic modification. Second, it naturally follows from neurophysiological and psychological research starting with Hebb's postulate [D. Hebb. (1949). The organization of behavior. Wiley, New York]. Third, it is simple, computationally cheap and its parameters are straightforward to determine. This rule is further extended by addition of four different types of gating derived from conventionally used types of gated decay in learning rules for continuous firing rate neural networks. The results show that the advantages of using these gatings are transferred to the new rule without sacrificing its dependency on spike-timing.

Dynamic projection network for supervised pattern classification

This paper describes the development of the utility of a dynamic neural network known as projection network for pattern classification. It first gives the derivation of the projection network, and then describes the network architecture and analyzes properties such as equilibrium points and their stability condition. The procedures for utilizing the projection network for pattern classification are established and the benefits are discussed. The proposed classification system is then tested with well-known benchmark data sets, namely the Fisher’s iris data, the heart disease data and the credit screening data and the results are compared to other classifiers including Neural Network Rule Base (NNRB), Genetic Algorithm Rule Base (GARB), Rough Set, and C4.5 decision tree. The projection network was proven to be a viable alternative to existing methods.

Identifying multiple abdominal organs from CT image series using a multimodule contextual neural network and spatial fuzzy rules

Identifying abdominal organs is one of the essential steps in visualizing organ structure to assist in teaching, clinical training, diagnosis, and medical image retrieval. However, due to partial volume effects, gray-level similarities of adjacent organs, contrast media affect, and the relatively high variations of organ position and shape, automatically identifying abdominal organs has always been a high challenging task. To conquer these difficulties, this paper proposes combining a multimodule contextual neural network and spatial fuzzy rules and fuzzy descriptors for automatically identifying abdominal organs from a series of CT image slices. The multimodule contextual neural network segments each image slice through a divide-and-conquer concept, embedded within multiple neural network modules, where the results obtained from each module are forwarded to other modules for integration, in which contextual constraints are enforced. With this approach, the difficulties arising from partial volume effects, gray-level similarities of adjacent organs, and contrast media affect can be reduced to the extreme. To address the issue of high variations in organ position and shape, spatial fuzzy rules and fuzzy descriptors are adopted, along with a contour modification scheme implementing consecutive organ region overlap constraints. This approach has been tested on 40 sets of abdominal CT images, where each set consists of about 40 image slices. We have found that 99% of the organ regions in the test images are correctly identified as its belonging organs, implying the high promise of the proposed method.

An approach to optimise the critical sensor locations in one-dimensional novel distributive tactile surface to maximise performance

The distributive approach to tactile sensing is a novel approach. The method relies on the distributed deformation of the surface in response to the applied load to a few sensing points within the surface area. The description of the contacting load is then interpreted into meaningful descriptors typically by using a neural network or fuzzy rules. The method has been shown to interpret descriptors such as load position, load value and load width and relies on strong coupling between the sensory data retrieved. This opposes the design aims in many alternative tactile sensing systems that formulate load description from isolated discrete data detected over an array of sensing elements, or that delineate force descriptions through a structure that minimises coupling on Cartesian axes. For distributive tactile sensors, the performance can be optimised through placement of sensing points such that the obtained information is optimal. This paper examines the effect on performance of sensor location points on an experimental one-dimensional surface designed for this purpose. The algorithm interpreting load descriptors was a back-propagation neural network. The critical parameter of sensor location is optimised using the genetic algorithm (GA) and principal component analysis (PCA) approach. It is shown that when an optimised configuration is used load position can be predicted to within 5% of the full range by using as few as two sensing elements, and that performance is improved by using additional sensor points. The results of this study are a basis for selecting sensor locations to achieve high performance with planar one- and two-dimensional distributive tactile surfaces.

Using Neural Network Rule Extraction and Decision Tables for Credit-Risk Evaluation

Credit-risk evaluation is a very challenging and important management science problem in the domain of financial analysis. Many classification methods have been suggested in the literature to tackle this problem. Neural networks, especially, have received a lot of attention because of their universal approximation property. However, a major drawback associated with the use of neural networks for decision making is their lack of explanation capability. While they can achieve a high predictive accuracy rate, the reasoning behind how they reach their decisions is not readily available. In this paper, we present the results from analysing three real-life credit-risk data sets using neural network rule extraction techniques. Clarifying the neural network decisions by explanatory rules that capture the learned knowledge embedded in the networks can help the credit-risk manager in explaining why a particular applicant is classified as either bad or good. Furthermore, we also discuss how these rules can be visualized as a decision table in a compact and intuitive graphical format that facilitates easy consultation. It is concluded that neural network rule extraction and decision tables are powerful management tools that allow us to build advanced and userfriendly decision-support systems for credit-risk evaluation.

Improved Training Rules for Multilayered Feedforward Neural Networks

We propose an improved supervisory training rule for multilayered feedforward neural networks (FNNs). The proposed method analytically estimates the optimal solutions for the output weights of FNNs. Also, using the optimal solutions, it reduces the searching space as much as the output weights in the iterative high-dimensional nonlinear optimization problem for the supervisory training. As a result, we can secure a much faster convergence rate and better robustness compared to the previous full-dimensional training rules.

The ups and downs of Hebb synapses.

Abstract Modelers have come up with many different learning rules for neural networks. When a teacher specifies the correct output, rules work better than pure Hebb rules in which the changes in synapse strength depend on the correlation between pre and postsynaptic activities. But for unsupervised learning, Hebb rules can be very effective if they are combined with suitable normalization or unlearning terms to prevent the synapses growing without bound. Hebb rules that use rates of change of activity instead of activity itself are useful for discovering perceptual invariants and may also provide a way of implementing learning. It would be truly wonderful if randomly connected neural networks could turn themselves into useful computing devices by using some simple rule to modify the strengths of synapses. This was the hope that lay behind the original Hebb learning rule and it is the vision that has driven neural network modelers for half a century. Initially, researchers tried simulating various rules to see what would happen. After a decade or two of messing around, researchers realized that there was a much better way to explore the space of possible learning rules: First write down an objective function (a quantitative definition of how well the network is performing) and then use elementary calculus to derive a learning rule that will improve the objective function. For the last few decades, the big theoretical advances in learning rules for neural networks have been associated with new optimization methods and new ideas about what objective function should be optimized. If we think of a neural network as a device for converting input vectors into output vectors, it is obvious that one sensible objective is to minimize some measure of the difference between the output the network actually produces and the output it ought to produce. This approach led to effective error-driven learning rules such as the Widrow-Hoff rule (Widrow & Hoff, 1960) and the perceptron convergence procedure (Rosenblatt, 1961) and it was later generalized to multilayer networks by using backpropagation of the errors to get training signals for intermediate hidden layers (Rumelhart, Hinton, & Williams, 1986). Within the neural network community, the approach of using the product of pre and postsynaptic activities to drive learning was seen as inferior to error-- driven methods that use the product of the presynaptic activity and the postsynaptic activity derivative - the rate at which the objective function changes as the postsynaptic activity is changed. Even when the task was merely to associate random input vectors with random output vectors, it was shown that an rule worked much better than a Hebbian rule. Unfortunately, learning has some serious drawbacks. It requires a teacher to specify the right answer and it is hard to see how neurons could implement the backpropagation required by multilayer versions. It is possible to get the teaching signal from the data itself by trying to predict the next term in a temporal sequence (Elman, 1991) or by trying to reconstruct the input data at the output (Hinton, 1989) but it is also possible to use quite different objective functions for learning. Some of these alternative objective functions lead to learning rules that are far more Hebbian in flavour. A common objective in processing high-dimensional data is to reduce the dimensionality without losing the ability to reconstruct the raw data from the reduced representation. If we measure the accuracy of the reconstruction by the squared error, the optimal strategy is to extract the principal components - the dominant directions of variation in the data. Oja (1982) showed how to extract the first principal component using Hebbian learning to maximize the squared output of a neuron combined with normalization of the synapse strengths to prevent them growing without bound. …

Intelligent Classification of the Drop Hammer Forming Process Method

Forging is a cost-effective way to produce net-shape or nearnet-shape components. Forged components are used throughout the manufacturing sector in many different applications. Owing to the lack of an accurate process model, modern forging operations still largely rely on operator skills and experience. The expertise of skilled operators can help to develop a forging process model. However, these operators’ expert knowledge is accumulated through years of hands-on experience and is often biased towards their own heuristic. It is well known that accurate acquisition of this type of knowledge is challenging and time-consuming. In this paper, an innovative approach is developed to acquire forging process knowledge automatically by combining learning ability of the neural networks with the structured knowledge representation of rule-based systems. The approach is applied to the classification of process methods used in a type of impression-die forging, namely, drop hammer forming. Specifically, process data from an aerospace company’s production facility are collected. The data are processed and then used to train a back-propagation neural network. By analysing the connections and weights of the trained neural network, concise and intelligible rules are extracted. These rules can be used to allow a clearer specification of the drop hammer forming process plan and to shorten learning curves for novice operators.

A neural network and fuzzy rule base hybrid for pattern classification

This paper presents a novel hybrid of the two complimentary technologies of soft computing viz. neural networks and fuzzy logic to design a fuzzy rule based pattern classifier for problems with higher dimensional feature spaces. The neural network component of the hybrid, which acts as a pre-processor, is designed to take care of the all-important issue of feature selection. To circumvent the disadvantages of the popular back propagation algorithm to train the neural network, a meta-heuristic viz. threshold accepting (TA) has been used instead. Then, a fuzzy rule based classifier takes over the classification task with a reduced feature set. A combinatorial optimisation problem is formulated to minimise the number of rules in the classifier while guaranteeing high classification power. A modified threshold accepting algorithm proposed elsewhere by the authors (Ravi V, Zimmermann H.-J. (2000) Eur J Oper Res 123: 16–28) has been employed to solve this optimization problem. The proposed methodology has been demonstrated for (1) the wine classification problem having 13 features and (2) the Wisconsin breast cancer determination problem having 9 features. On the basis of these examples the results seem to be very interesting, as there is no reduction in the classification power in either of the problems, despite the fact that some of the original features have been completely eliminated from the study. On the contrary, the chosen features in both the problems yielded 100% classification power in some cases.

Supervised and unsupervised learning with two sites of synaptic integration.

Many learning rules for neural networks derive from abstract objective functions. The weights in those networks are typically optimized utilizing gradient ascent on the objective function. In those networks each neuron needs to store two variables. One variable, called activity, contains the bottom-up sensory-fugal information involved in the core signal processing. The other variable typically describes the derivative of the objective function with respect to the cell's activity and is exclusively used for learning. This variable allows the objective function's derivative to be calculated with respect to each weight and thus the weight update. Although this approach is widely used, the mapping of such two variables onto physiology is unclear, and these learning algorithms are often considered biologically unrealistic. However, recent research on the properties of cortical pyramidal neurons shows that these cells have at least two sites of synaptic integration, the basal and the apical dendrite, and are thus appropriately described by at least two variables. Here we discuss whether these results could constitute a physiological basis for the described abstract learning rules. As examples we demonstrate an implementation of the backpropagation of error algorithm and a specific self-supervised learning algorithm using these principles. Thus, compared to standard, one-integration-site neurons, it is possible to incorporate interesting properties in neural networks that are inspired by physiology with a modest increase of complexity.

A comparison between two neural network rule extraction techniques for the diagnosis of hepatobiliary disorders

Neural networks have been widely used as tools for prediction in medicine. We expect to see even more applications of neural networks for medical diagnosis as recently developed neural network rule extraction algorithms make it possible for the decision process of a trained network to be expressed as classification rules. These rules are more comprehensible to a human user than the classification process of the networks which involves complex nonlinear mapping of the input data. This paper reports the results from two neural network rule extraction techniques, NeuroLinear and NeuroRule applied to the diagnosis of hepatobiliary disorders. The dataset consists of nine measurements collected from patients in a Japanese hospital and these measurements have continuous values. NeuroLinear generates piece-wise linear discriminant functions for this dataset. The continuous measurements have previously been discretized by domain experts. NeuroRule is applied to the discretized dataset to generate symbolic classification rules. We compare the rules generated by the two techniques and find that the rules generated by NeuroLinear from the original continuously valued dataset to be slightly more accurate and more concise than the rules generated by NeuroRule from the discretized dataset.