Standard Error Back-propagation Algorithm Research Articles

A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks.

Spiking neural networks (SNNs) represent the most prominent biologically inspired computing model for neuromorphic computing (NC) architectures. However, due to the nondifferentiable nature of spiking neuronal functions, the standard error backpropagation algorithm is not directly applicable to SNNs. In this work, we propose a tandem learning framework that consists of an SNN and an artificial neural network (ANN) coupled through weight sharing. The ANN is an auxiliary structure that facilitates the error backpropagation for the training of the SNN at the spike-train level. To this end, we consider the spike count as the discrete neural representation in the SNN and design an ANN neuronal activation function that can effectively approximate the spike count of the coupled SNN. The proposed tandem learning rule demonstrates competitive pattern recognition and regression capabilities on both the conventional frame- and event-based vision datasets, with at least an order of magnitude reduced inference time and total synaptic operations over other state-of-the-art SNN implementations. Therefore, the proposed tandem learning rule offers a novel solution to training efficient, low latency, and high-accuracy deep SNNs with low computing resources.

Combining experimental design and artificial neural networks for the determination of chlorinated compounds in fish using matrix solid-phase dispersion

The purpose of this contribution is to present a very effective strategy for the development of matrix solid phase dispersion (MSPD) extraction methodology for the determination of chlorinated compounds in fish using experimental design methods and artificial neural networks (ANNs). The MSPD extraction method of compounds is a preparation method that comprises sample homogenization, cellular disruption, fractionation and purification in a single process. Many parameters have to be taken care of when developing an MSPD extraction method because its performance is mainly affected by column packing and elution procedure. In this study, the best possible performance of MSPD has been achieved using experimental design and ANN modeling. The ANN used is a multilayer perceptron (MLP) trained with the standard error back propagation algorithm. Experimental results demonstrate that the proposed soft computing strategy is very effective, efficient and achieves very satisfactory results.

A hybrid approach for processing parameters optimization of Ti-22Al-25Nb alloy during hot deformation using artificial neural network and genetic algorithm

In the present investigation, isothermal compression tests of Ti-22Al-25Nb alloy were carried out under various hot deformation conditions, including the deformation temperature range of 940–1060 °C and the strain rate range of 0.01–10 s −1. The constitutive relationship of Ti-22Al-25Nb alloy was developed using artificial neural network (ANN). During training process, standard error back-propagation algorithm was employed in the network model using experimental data sets. Based on the fitness function obtained from established ANN model, the optimization model of hot processing parameters for Ti-22Al-25Nb alloy was successfully created using genetic algorithm (GA). The optimal results achieved from the integrated ANN and GA optimization model were tested by using processing map. Consequently, it can be suggested that the combined approach of ANN and GA provides a novel way with respect to the optimization of processing parameters in the field of materials science.

\title{Connectionist model of action learning and naming}

\title{Connectionist model of action learning and naming}

New Dynamical Optimal Learning for Linear Multilayer FNN

This letter presents a new dynamical optimal learning (DOL) algorithm for three-layer linear neural networks and investigates its generalization ability. The optimal learning rates can be fully determined during the training process. The mean squared error (mse) is guaranteed to be stably decreased and the learning is less sensitive to initial parameter settings. The simulation results illustrate that the proposed DOL algorithm gives better generalization performance and faster convergence as compared to standard error back propagation algorithm.

Supervised training algorithms for B-Spline neural networks and neuro-fuzzy systems

Complete supervised training algorithms for B-Spline neural networks and fuzzy rule-based systems are discussed. By introducing the relationships between B-Spline neural networks and Mamdani (satisfying certain assumptions) and Takagi-Kang-Sugeno fuzzy models, training algorithms developed initially for neural networks can be adapted to fuzzy systems. The standard training criterion is reformulated, by separating its linear and nonlinear parameters. By employing this reformulated criterion with the Levenberg-Marquardt algorithm, a new training method, offering a fast rate of convergence is obtained. It is also shown that the standard Error-Back Propagation algorithm, the most common training method for this class of systems, exhibits a very poor and unreliable performance.

Completely Supervised Training Algorithms for B-spline Neural Networks and Neuro-fuzzy Systems

ABSTRACT Complete supervised training algorithms tor B-spline neural networks and fuzzy rule-based systems are discussed. By introducing the relationships between B-spline neural networks and certain types of fuzzy models, training algorithms developed initially for neural networks can be adapted to fuzzy systems. The standard training criterion is reformulated, by separating its parameters into linear and non linear ones. By employing this reformulation with the Levenberg-Marquardt algorithm, a new training method. offering a fast rate or convergence, and a robust performance compared with the standard Error-Back Propagation algorithm.

Adaptive learning algorithms to incorporate additional functional constraints into neural networks

In this paper, adaptive learning algorithms to obtain better generalization performance are proposed. We specifically designed cost terms for the additional functionality based on the first- and second-order derivatives of neural activation at hidden layers. In the course of training, these additional cost functions penalize the input-to-output mapping sensitivity and high-frequency components in training data. A gradient-descent method results in hybrid learning rules to combine the error back-propagation, Hebbian rules, and the simple weight decay rules. However, additional computational requirements to the standard error back-propagation algorithm are almost negligible. Theoretical justifications and simulation results are given to verify the effectiveness of the proposed learning algorithms.

River flow prediction using artificial neural networks: generalisation beyond the calibration range

Artificial neural networks (ANNs) provide a quick and flexible means of creating models for river flow prediction, and have been shown to perform well in comparison with conventional methods. However, if the models are trained using a dataset that contains a limited range of values, they may perform poorly when encountering events containing previously unobserved values. This failure to generalise limits their use as a tool in applications where the data available for calibration is unlikely to cover all possible scenarios.This paper presents a method for improved generalisation during training by adding a guidance system to the cascade-correlation learning architecture. Two case studies from catchments in the UK are prepared so that the validation data contains values that are greater or less than any included in the calibration data. The ability of the developed algorithm to generalise on new data is compared with that of the standard error backpropagation algorithm. The ability of ANNs trained with different output activation functions to extrapolate beyond the calibration data is assessed.

Dynamics of compartmental model neurons

Recent developments in the dynamics of compartmental model neurons are described. In particular, an explicit analytical expression for the response function of a neuron with arbitrary dendritic tree structure is presented, which provides a general framework for exploring the effects of complex geometries on spatiotemporal pattern processing in neurons. Some examples of how compartmental structure can enhance a neuron's sensitivity to temporal features of an input are given. Shunting contributions are included at a perturbative level, and are shown to have an important influence on the time constants of a neuron's response. The effects of synaptic background activity are also discussed. A solution to the dynamical equations for a compartmental model with somatic potential reset is presented that takes proper account of the electrical coupling between soma and dendrites. Finally, some applications of this work to artificial neural networks are indicated, including an extension of the standard error back propagation algorithm to the case of compartmental neurons with known response function.

Maximum mutual information neural networks for hybrid connectionist-HMM speech recognition systems

This paper proposes a novel approach for a hybrid connectionist-hidden Markov model (HMM) speech recognition system based on the use of a neural network as vector quantizer. The neural network is trained with a new learning algorithm offering the following innovations. (1) It is an unsupervised learning algorithm for perceptron-like neural networks that are usually trained in the supervised mode. (2) Information theory principles are used as learning criteria, making the network especially suitable for combination with a HMM-based speech recognition system. (3) The neural network is not trained using the standard error-backpropagation algorithm but using instead a newly developed self-organizing learning approach. The use of the hybrid system with the neural vector quantizer results in a 25% error reduction compared with the same HMM system using a standard k-means vector quantizer. The training algorithm can be further refined by using a combination of unsupervised and supervised learning algorithms. Finally, it is demonstrated how the new learning approach can be applied to multiple-feature hybrid speech recognition systems, using a joint information theory-based optimization procedure for the multiple neural codebooks, resulting in a 30% error reduction.