Classical Backpropagation Algorithm Research Articles

An adaptive backpropagation neural network for real-time ischemia episodes detection: development and performance analysis using the European ST-T database.

A supervised neural network (NN)-based algorithm was used for automated detection of ischemic episodes resulting from ST segment elevation or depression. The performance of the method was measured using the European ST-T database. In particular, the performance was measured in terms of beat-by-beat ischemia detection and in terms of the detection of ischemic episodes. The algorithm used to train the NN was an adaptive backpropagation (BP) algorithm. This algorithm drastically reduces training time (tenfold decrease in our case) when compared to the classical BP algorithm. The recall phase of the NN is then extremely fast, a fact that makes it appropriate for real-time detection of ischemic episodes. The resulting NN is capable of detecting ischemia independent of the lead used. It was found that the average ischemia episode detection sensitivity is 88.62% while the ischemia duration sensitivity is 72.22%. The results show that NN can be used in electrocardiogram (ECG) processing in cases where fast and reliable detection of ischemic episodes is desired as in the case of critical care units (CCU's).

A successive overrelaxation backpropagation algorithm for neural-network training

A variation of the classical backpropagation algorithm for neural network training is proposed and convergence is established using the perturbation results of Mangasarian and Solodov. The algorithm is similar to the successive overrelaxation (SOR) algorithm for systems of linear equations and linear complementary problems in using the most recently computed values of the weights to update the values on the remaining arcs.

Sensor validation using hardware-based on-line learning neural networks

The objective of this document Is to show the capabilities of parallel hardware-based on-line learning neural networks (NNs). This specific application is related to an on-line estimation problem for sensor validation purposes. Neural-network-based microprocessors are starting to be commercially available. However, most of them feature a learning performed with the classic back-propagation algorithm (BPA). To overcome this lack of flexibility a customized motherboard with transputers was implemented for this investigation, The extended BPA (EBPA), a modified and more effective BPA, was used for the on-line learning, These parallel hardware-based neural architectures were used to implement a sensor failure detection, identification, and accommodation scheme in the model of a night control system assumed to be without physical redundancy in the sensory capabilities. The results of this study demonstrate the potential for these neural schemes for implementation in actual flight control systems of modern high performance aircraft, taking advantage of the characteristics of the extended back-propagation along with the parallel computation capabilities of NN customized hardware.

A new system identification technique using Kaiman filtering and multilayer neural networks

The objective of this work is to use the back-propagation algorithm in conjunction with Kaiman filtering in order to establish a new self-learning technique of multilayer neural network (MNN). This new technique is developed by directly building a Kaiman filtering model for each perceptron in order to increase the adaptability of the MNN and to provide for on-line nonlinear system identification. We demonstrate that this new technique is faster and more stable than the classical back-propagation algorithm for training multilayer perceptrons. We also find that it is less sensitive to the initial weights and to the learning parameters.

Efficient segmentation of large-area skin images: an overview of image processing.

This paper is the first of a series describing research and development performed by the SPOTS project towards automating the process of monitoring pigmented lesions for change over time. A comprehensive overview of digital image processing techniques used for the systematic approach to segmentation of large-area skin images, including the description and classification of skin features, is presented. The multiresolution hierarchical segmentation technique, which makes use of image pyramid data structures and cooperative computation with unforced linking, was used to group pixels with similar color properties. This technique was further enhanced to enforce contiguity of segments by determining connected components. The resulting segmentation was then classified by using representative properties of pyramid nodes as input to a cascade of multilayer perceptron neural networks trained using the classical backpropagation algorithm. In order to make our system more resistant to imaging distortions, a technique was developed to grow regions based upon the resulting classification. This system produces a segmentation at multiple resolutions based upon color, where each segment is a contiguous group of pixels on the image. The classification makes use of both spatial and color properties, as well as parent-child relationships within the pyramid. The addition of classification-driven region growing reduces the change in lesion area where the boundary fades into surrounding skin. Furthermore, the system contains parameters that can be tuned to provide desirable results.

A HYBRID LEARNING RULE FOR A FEEDFORWARD NETWORK

In feedforward networks trained by the classic backpropagation algorithm, as introduced by Rumelhart et al., the weights are modified according to the method of steepest descent. The goal of this weight modification is to minimise the error in network-outputs for a given training set. Basing upon Jacobs’ work, we point out drawbacks of steepest descent and suggest improvements on it. These yield a feedforward network, which adjusts its weights according to a (globally convergent) parallel coordinate descent method. Then we combine this parallel coordinate descent with a hybrid learning rule from [Formula: see text]-rule and momentum version. For adjusting the parameters of this rule we use a Sugeno/Tagaki fuzzy controller. We only need four rules for the controlling process. Because of using the Sugeno/Takagi controller, no defuzzification has to be performed and the controller works faster than the familiar one. We conclude that this algorithm is very suitable for fast training and global convergence.

A Class of Recurrent Neural Networks for Adaptive Control of Nonlinear Systems

In this paper, recurrent neural networks (RNN) are used to construct nonlinear learning systems for a class of unknown nonlinear systems. We describe the RNN controllers with reference to adaptive control theory and put emphasis on their flexible structure to express nonlinear systems. The proposed control scheme employs two types of RNN: The first network learns the unknown dynamics of the nonlinear process to be controlled. The second utilises the information given by the first network about the process, to learn the inverse dynamics of the neural network process model such that the error between the system and the desired output is nullified. We also present a generalized criterion for training the RNN which gives rise to new control schemes. The training methodology that is used for the weights adaptation is much faster than classical backpropagation algorithm. We investigate the effects the type of nonlinear function, present in the generalized criterion, have on performance of the learning algorithm of unknown nonlinear system dynamics.

Deterministic global optimal FNN training algorithms

To avoid local minimum solutions in back propagation learning, we propose to treat feedforward neural network training as a global optimization problem. In particular, we considered using branch-and-bound based Lipschitz optimization methods in neural network training, and developed globally optimal training algorithms (GOTA). The standard criterion function of a feedforward neural network is Lipschitzian. The effectiveness of GOTA is improved by using dynamically computed local Lipschitz constants over subsets of the weight space. Local search procedures, such as the classic back propagation algorithm, can be incorporated in GOTA. The local search-augmented global algorithms improve the learning efficiency of GOTA while retaining the globally convergent property.

Learning Algorithms for Neural-Net Decision Support

Artificial Neural Networks present a new paradigm for decision support that is adaptive and capable of integrating knowledge acquisition, problem solving, and learning. In this paper, we show that the performance of the classical back-propagation algorithm for training artificial neural networks can be improved by applying modified approaches which have better convergence properties, and are faster than the commonly used steepest gradient search method. Simulation results on test problems show that the proposed algorithms perform better than the steepest gradient search algorithm in terms of faster convergence with learning accuracy at least as good as in classical back propagation. Inductive learning algorithms (ID3 and NEWQ) and Probit are also used to compare the relative performances of connectionist, inductive learning and statistical procedures. INFORMS Journal on Computing, ISSN 1091-9856, was published as ORSA Journal on Computing from 1989 to 1995 under ISSN 0899-1499.

Robustness in Multilayer Perceptrons

In this paper, we study the robustness of multilayer networks versus the destruction of neurons. We show that the classical backpropagation algorithm does not lead to optimal robustness and we propose a modified algorithm that improves this capability.

Inter-Block Redundancy Reduction in Vector-Quantized Images by a Neural Predictor

A prediction scheme is proposed, which allows one to exploit residual interblock correlation after vector quantization (VQ), without loss in the quality of reconstructed images. Such a scheme is based on a Perceptron neural network, whose goal is to produce an estimation of a block, once the neighboring blocks have already been coded. The network is trained on a set of images by means of the classical back-propagation algorithm. The error to be minimized is the Euclidean distance (i.e., the mean square error, MSE) between the block generated by the network (using the four previously-coded neighboring blocks as input) and the current block. During the encoding phase, for each block vector-quantized, the block produced by the network is used to generate a reduced codebook that, in the case of a correct prediction, allows one to use a smaller number of bits for the transmission of the block. Experimental results show that the probability of a correct prediction ranges from 60 to 75 %, thus producing an increase in compression by a factor of 1.3 to 1.6. The main advantages of the presented approach lie in the efficiency of the neural predictor (even for images not included in the training set), and in reduced memory requirements in comparison with other methods previously proposed.

Parallel recursive prediction error algorithm for training layered neural networks

A new recursive prediction error algorithm is derived for the training of feedforward layered neural networks. The algorithm enables the weights in each neuron of the network to be updated in an efficient parallel manner and has better convergence properties than the classical back propagation algorithm. The relationship between this new parallel algorithm and other existing learning algorithms is discussed. Examples taken from the fields of communication channel equalization and nonlinear systems modelling are used to demonstrate the superior performance of the new algorithm compared with the back propagation routine.