Classical Backpropagation Research Articles

Comparing a genetic fuzzy and a neurofuzzy classifier for credit scoring

In this paper, we evaluate and contrast two types of fuzzy classifiers for credit scoring. The first classifier uses evolutionary optimization and boosting for learning fuzzy classification rules. The second classifier is a fuzzy neural network that employs a fuzzy variant of the classic backpropagation learning algorithm. The experiments are carried out on a real life credit scoring data set. It is shown that, for the case at hand, the boosted genetic fuzzy classifier performs better than both the neurofuzzy classifier and the well-known C4.5(rules) decision tree(rules) induction algorithm. However, the better performance of the genetic fuzzy classifier is offset by the fact that it infers approximate fuzzy rules which are less comprehensible for humans than the descriptive fuzzy rules inferred by the neurofuzzy classifier.

Natural gradient learning neural networks for adaptive inversion of Hammerstein systems

This letter applies natural gradient (NG) learning neural networks for adaptive inversion of Hammerstein systems. The system model is composed of a memoryless nonlinearity g(.) followed by a linear filter H. The inverse system is modeled by a neural network composed of an adaptive filter Q followed by a memoryless nonlinear perceptron. The adaptive filter Q aims at inverting the linear part of the system (adaptive deconvolution). The perceptron aims at inverting the memoryless function (adaptive function inversion). The adaptive system is trained using the NG descent algorithm. The letter shows through computer simulations that the NG approach outperforms the classical backpropagation algorithm in terms of mean-squared-error performance and convergence speed.

On-line fault section estimation in power systems with radial basis function neural network

Fault section estimation is of great importance to the restoration of power systems. Many techniques have been used to solve this problem. In this paper, the application of radial basis function neural network (RBF NN) to fault section estimation is addressed. The orthogonal least square (OLS) algorithm has been extended to optimize the number of neurons in hidden layer and the connection weights of RBF NN. A classical back-propagation neural network (BP NN) has been developed to solve the same problem for comparison. Computer test is conducted on a four-bus test system and the test results show that the RBF NN is quite effective and superior to BP NN in fault section estimation.

STOCHASTIC APPROACHES TO DYNAMIC NEURAL NETWORK TRAINING. ACTUATOR FAULT DIAGNOSIS STUDY

A paper deals with application of stochastic methods for dynamic neural network training. The considered network is composed of dynamic neurons, which contain inner feedbacks. This network can be used as a part of a fault diagnosis system to generate residuals. Up-to-date training algorithms, based on the classical back propagation, suffer from entrapment in local minima of an error function. Two stochastic algorithms are tested as training algorithms to overcome these difficulties. Efficiency of the proposed learning methods is checked using data recorded at Lublin Sugar Factory, Poland.

A use of a neural network to evaluate contrast enhancement curves in breast magnetic resonance images.

For the diagnosis of breast cancer using magnetic resonance imaging (MRI), one of the most important parameters is the analysis of contrast enhancement. A three-dimensional MR sequence is applied before and five times after bolus injection of paramagnetic contrast medium (Gd-DTPA). The dynamics of absorption are described by a time/intensity enhancement curve, which reports the mean intensity of the MR signal in a small region of interest (ROI) for about 8 minutes after contrast injection. The aim of our study was to use an artificial neural network to automatically classify the enhancement curves as "benign" or "malignant." We used a classic feed-forward back-propagation neural network, with three layers: five input nodes, two hidden nodes, and one output node. The network has been trained with 26 pathologic curves (10 invasive carcinoma [K], two carcinoma-in-situ [DCIS], and 14 benign lesion [B]). The trained network has been tested with 58 curves (36 K, one DCIS, 21 B). The network was able to correctly identify the test curves with a sensitivity of 76% and a specificity of 90%. For comparison, the same set of curves was analyzed separately by two radiologists (a breast MR expert and a resident radiologist). The first correctly interpreted the curves with a sensitivity of 76% and a specificity of 90%, while the second scored 59% for sensitivity and 90% for specificity. These results demonstrate that a trained neural network recognizes the pathologic curves at least as well as an expert radiologist. This algorithm can help the radiologist attain rapid and affordable screening of a large number of ROIs. A complete automatic computer-aided diagnosis support system should find a number of potentially interesting ROIs and automatically analyze the enhancement curves for each ROI by neural networks, reporting to the radiologist only the potentially pathologic ROIs for a more accurate, manual, repeated evaluation.

A HYBRID APPROACH TO KNOWLEDGE ACQUISITION USING NEURAL PETRI NETS AND DS THEORY

Knowledge acquisition from multiple experts and its refinement are important issues in knowledge management of an expert system. The paper presents a novel approach to handling the above problems by combining the synergistic behavior of neural Petri nets and the Dempster–Shafer theory. The Dempster–Shafer theory has been employed here to reduce the scope of uncertainty in the supplied noisy input instances and the inferences generated therefrom by the multiple experts. The noise-free training instances thus obtained are subsequently used to train the neural Petri net model for refining the parameters of its knowledge base. A comparison of the performance of the proposed training algorithm with the classical back-propagation algorithm has also been presented in the paper.

Fault detection and isolation in robotic manipulators via neural networks: A comparison among three architectures for residual analysis

AbstractIn this article we discuss artificial neural networks‐based fault detection and isolation (FDI) applications for robotic manipulators. The artificial neural networks (ANNs) are used for both residual generation and residual analysis. A multilayer perceptron (MLP) is employed to reproduce the dynamics of the robotic manipulator. Its outputs are compared with actual position and velocity measurements, generating the so‐called residual vector. The residuals, when properly analyzed, provides an indication of the status of the robot (normal or faulty operation). Three ANNs architectures are employed in the residual analysis. The first is a radial basis function network (RBFN) which uses the residuals of position and velocity to perform fault identification. The second is again an RBFN, except that it uses only the velocity residuals. The third is an MLP which also performs fault identification utilizing only the velocity residuals. The MLP is trained with the classical back‐propagation algorithm and the RBFN is trained with a Kohonen self‐organizing map (KSOM). We validate the concepts discussed in a thorough simulation study of a Puma 560 and with experimental results with a 3‐joint planar manipulator. © 2001 John Wiley & Sons, Inc.

Estimating Labor Productivity Using Probability Inference Neural Network

This paper discusses the derivation of a probabilistic neural network classification model and its application in the construction industry. The probability inference neural network (PINN) model is based on the same concepts as those of the learning vector quantization method combined with a probabilistic approach. The classification and prediction networks are combined in an integrated network, which required the development of a different training and recall algorithm. The topology and algorithm of the developed model was presented and explained in detail. Portable computer software was developed to implement the training, testing, and recall for PINN. The PINN was tested on real historical productivity data at a local construction company and compared to the classic feedforward back-propagation neural network model. This showed marked improvement in performance and accuracy. In addition, the effectiveness of PINN for estimating labor production rates in the context of the application domain was validat...

EEG-based discrimination between imagination of left and right hand movements using adaptive gaussian representation

This article uses the Adaptive Gaussian Representation (AGR) for human electroencephalogram (EEG) feature extraction aiming the discrimination among mental tasks to be used in a brain computer interface (BCI). It does not focus on the AGR time–frequency representation, but rather on their projection coefficients. Ten volunteers were asked to imagine either right or left hand movement, according to a proper visual stimulus. The features of the resulting EEG signals were characterised by extracting AGR coefficients. Classification was carried out using a Multilayer perceptron (MLP) trained with the classical backpropagation algorithm. Overall results show that AGR coefficients representation is able to reveal a significant EEG discrimination between imagination of right and left hand movement with a mean classification performance of 91%±5.8% achieved for female subjects and 87%±5.0% achieved for male subjects.

Fuzzy neural networks for estimation and fuzzy controller design: simulation study for a pulp batch digester

Abstract A structural implementation of a fuzzy inference system through connectionist network based on MLP with logical neurons connected through binary and numerical weights is considered. The resulting fuzzy neural network is trained using classical backpropagation to learn the rules of inference of a fuzzy system, by adjustment of the numerical weights. For controller design, training is carried out off line in a closed loop simulation. Rules for the fuzzy logic controller are extracted from the network by interpreting the consequence weights as measure of confidence of the underlying rule. The framework is used in a simulation study for estimation and control of a pulp batch digester. The controlled variable, the Kappa number, a measure of lignin content in the pulp, which is not measurable is estimated through temperature and liquor concentration using the fuzzy neural network. On the other hand a fuzzy neural network is trained to control the Kappa number and rules are extracted from the trained network to construct a fuzzy logic controller.

An adaptive backpropagation neural network for real-time ischemia episodes detection: development and performance analysis using the European ST-T database.

A supervised neural network (NN)-based algorithm was used for automated detection of ischemic episodes resulting from ST segment elevation or depression. The performance of the method was measured using the European ST-T database. In particular, the performance was measured in terms of beat-by-beat ischemia detection and in terms of the detection of ischemic episodes. The algorithm used to train the NN was an adaptive backpropagation (BP) algorithm. This algorithm drastically reduces training time (tenfold decrease in our case) when compared to the classical BP algorithm. The recall phase of the NN is then extremely fast, a fact that makes it appropriate for real-time detection of ischemic episodes. The resulting NN is capable of detecting ischemia independent of the lead used. It was found that the average ischemia episode detection sensitivity is 88.62% while the ischemia duration sensitivity is 72.22%. The results show that NN can be used in electrocardiogram (ECG) processing in cases where fast and reliable detection of ischemic episodes is desired as in the case of critical care units (CCU's).

A fast training algorithm for neural networks

The recursive least squares method (RLS) is derived for the learning of multilayer feedforward neural networks. Simulation results on the XOR, 4-2-4 encoder, and function approximation problems indicate a fast learning process in comparison to the classical and momentum backpropagation (BP) algorithms.

A successive overrelaxation backpropagation algorithm for neural-network training

A variation of the classical backpropagation algorithm for neural network training is proposed and convergence is established using the perturbation results of Mangasarian and Solodov. The algorithm is similar to the successive overrelaxation (SOR) algorithm for systems of linear equations and linear complementary problems in using the most recently computed values of the weights to update the values on the remaining arcs.

Sensor validation using hardware-based on-line learning neural networks

The objective of this document Is to show the capabilities of parallel hardware-based on-line learning neural networks (NNs). This specific application is related to an on-line estimation problem for sensor validation purposes. Neural-network-based microprocessors are starting to be commercially available. However, most of them feature a learning performed with the classic back-propagation algorithm (BPA). To overcome this lack of flexibility a customized motherboard with transputers was implemented for this investigation, The extended BPA (EBPA), a modified and more effective BPA, was used for the on-line learning, These parallel hardware-based neural architectures were used to implement a sensor failure detection, identification, and accommodation scheme in the model of a night control system assumed to be without physical redundancy in the sensory capabilities. The results of this study demonstrate the potential for these neural schemes for implementation in actual flight control systems of modern high performance aircraft, taking advantage of the characteristics of the extended back-propagation along with the parallel computation capabilities of NN customized hardware.

Double exponential transform and generalized Hopfield network for modelling dynamic non-linear circuits

In this paper a new tool, i.e. the double exponential transform, is introduced for characterizing a non-linear dynamic circuit directly on the basis of its elements. The resulting characterization aims to improve the computational cost connected with the analysis and to add flexibility to the identification procedure of unknown elements in a circuit of this type. The identification is performed by using a generalized Hopfield network with modifiable activation functions. The proposed neural network deserves attention for two reasons, (i) It represents the model of the physical mechanism which maps input excitations to output responses via a circuit description of an electrical context. From this point of view it can be considered as a macromodel, in contrast with other neural networks yielding a micromodel of this context. (ii) It allows the identification of a device present in the circuit using what is known and focusing attention only on the unknown elements. The training of the network is carried out on its feedforward version by using the classical back-propagation rule. Afterwards the parameters are frozen and the recurrent operation is resumed. A simple circuit illustrates in detail all the aspects of the proposed method. In order to reduce the computational cost, the fast version of Fourier transform is used in connection with the double exponential transform. This requires one to know the ranges of values spanned by electrical quantities applied to the non-linear elements. These ranges are directly defined by the training set. © 1997 by John Wiley & Sons, Ltd.

Intelligent Control of Single Inverted Pendulum

The paper deals with neuro-fuzzy tools for non-linear system control. Inverted pendulum was chosen as a typical non-linear model for control. The control task was defined as lifting inverted pendulum from up-side-down position into standing position by a cart movement and keep this position of the pendulum stable. Two different controllers were observed, ANFIS (Adaptive Neural Inference System) and NARA (Neural Approximate Reasoning Approach). Finally a simple neuro-controller based on feed-forward neural network with classical back-propagation adaptation was studied. Results show that ANFIS seems to be a powerful tool for this kind of task. In the paper an extensive report about experiments with pendulum control is presented.

δ-NARMA neural networks: a new approach to signal prediction

This paper presents a new connectionist architecture for stochastic univariate signal prediction. After a review of related statistical and connectionist models pointing out their advantages and limitations, we introduce the /spl epsiv/-NARMA model as the simplest nonlinear extension of ARMA models. These models then provide the units of a MLP-like neural network: the /spl delta/-NARMA neural network. The associated learning algorithm is based on an extension of classical backpropagation and on the concept of virtual error. Such networks can be seen as an extension of ARIMA and ARARMA models and face the problem of nonstationary signal prediction. A theoretical study brings understanding of experimental phenomena observed during the /spl delta/-NARMA learning process. The experiments carried out on three railroad-related real-life signals suggest that /spl delta/-NARMA networks outperform other studied univariate models.

A new system identification technique using Kaiman filtering and multilayer neural networks

The objective of this work is to use the back-propagation algorithm in conjunction with Kaiman filtering in order to establish a new self-learning technique of multilayer neural network (MNN). This new technique is developed by directly building a Kaiman filtering model for each perceptron in order to increase the adaptability of the MNN and to provide for on-line nonlinear system identification. We demonstrate that this new technique is faster and more stable than the classical back-propagation algorithm for training multilayer perceptrons. We also find that it is less sensitive to the initial weights and to the learning parameters.

Adaptive internal representation in circular back-propagation networks

Surface- and prototype-based models are often regarded as alternative paradigms to represent internal knowledge in trained neural networks. This paper analyses a network model (Circular Back-Propagation) that overcomes such dualism by choosing the best-fitting representation adaptively. The model involves a straightforward modification to classical feed-forward structures to let neurons implement hyperspherical boundaries; as a result, it exhibits a notable representation power, and benefits from the simplicity and effectiveness of classical back-propagation training. Artificial testbeds support the model definition by demonstrating its basic properties; an application to a real, complex problem in the clinical field shows the practical advantages of the approach.

The robustness of canopy gap fraction estimates from red and near-infrared reflectances: A comparison of approaches

The estimation of canopy gap fraction [P o(0)] from red and near-infrared reflectance is investigated. Several approaches are compared, using classical vegetation indices ( VI) and backpropagation neural networks. The parameters of the VI-P o(0) relationships and the synaptic weights and biases of two-layer neural networks were successfully adjusted on an experimental data set and on data derived from radiative transfer model simulations. Three experimental data sets were acquired over sugar beet canopies. They expressed a large range of canopy architecture and soil background reflectance. Two of them that correspond to similar experimental procedures were merged together and then randomly split into two subsets: one for fitting the parameters of the VI-P o(0) relationships or to train the neural network, and the other for the evaluation of the predictive performances. The third experiment is used as an additional independent data set to test the robustness of the approaches. The model-simulated data set was generated using the SAIL and PROSPECT radiative transfer models, with input parameter values that were chosen to have similar distributions as observed over sugar beet canopies. As for the experimental data set, the model-simulated data set was split into a training (calibration) and a test (validation) data set. Results show that the gap fraction can be accurately estimated from the red and near-infrared reflectance without any external information except maybe the crop type (here sugar beet) and the soil line characteristics required for some of the vegetation indices. The best predictive performances were observed for the SAVI-like vegetation indices ( SAVI, TSAVI, MSAVI) and the poorest for the NDVI, PVI, and GEMI having intermediate although satisfactory results. Neural networks trained on the simulated data set appeared to be the most robust approach. It allows us to implicitly incorporate our knowledge about the physics of the radiative transfer in the interpretation of remote sensing data.