Classical Backpropagation Algorithm Research Articles

Modified Backpropagation Algorithm with Multiplicative Calculus in Neural Networks

Backpropagation is one of the most widely used algorithms for training feedforward deep neural networks. The algorithm requires a differentiable activation function and it performs computations of the gradient proceeding backwards through the feedforward deep neural network from the last layer through to the first layer. In order to calculate the gradient at a specific layer, the gradients of all layers are combined via the chain rule of calculus. One of the biggest disadvantages of the backpropagation is that it requires a large amount of training time. To overcome this issue, this paper proposes a modified backpropagation algorithm with multiplicative calculus. Multiplicative calculus provides an alternative to the classical calculus and it defines new kinds of derivative and integral forms in multiplicative form rather than addition and subtraction forms. The performance analyzes are discussed in various case studies and the results are given comparatively with classical backpropagation algorithm. It is found that the proposed modified backpropagation algorithm converges in less time to the solution and thus provides fast training in the given case studies. It is also shown that the proposed algorithm avoids the local minima problem.

Graphical multispectral radiation temperature inversion algorithm based on deep learning.

Neural networks are the most promising tool to solve the problem that an assumed emissivity model is needed in the field of multispectral radiometric temperature measurement. Existing neural network multispectral radiometric temperature measurement algorithms have been investigating the problems of network selection, network porting, and parameter optimization. The inversion accuracy and adaptability of the algorithms have been unsatisfactory. In view of the great success of deep learning in the field of image processing, this Letter proposes the idea of converting one-dimensional multispectral radiometric temperature data into two-dimensional image data for data processing to improve the accuracy and adaptability of multispectral radiometric temperature measurement by deep learning algorithms. Simulation and experimental validation are carried out. In the simulation, the error is less than 0.71% without noise and 1.80% with 5% random noise, which improves the accuracy by more than 1.55% and 2.66% compared with the classical BP (backpropagation) algorithm, and 0.94% and 0.96% compared with the GIM-LSTM (generalized inverse matrix-long short-term memory) algorithm. In the experiment, the error is less than 0.83%. This indicates that the method has high research value and is expected to lead multispectral radiometric temperature measurement technology to a new level.

A Dendritic Neuron Model Optimized by Meta-Heuristics with a Power-Law-Distributed Population Interaction Network for Financial Time-Series Forecasting

The famous McCulloch–Pitts neuron model has been criticized for being overly simplistic in the long term. At the same time, the dendritic neuron model (DNM) has been shown to be effective in prediction problems, and it accounts for the nonlinear information-processing capacity of synapses and dendrites. Furthermore, since the classical error back-propagation (BP) algorithm typically experiences problems caused by the overabundance of saddle points and local minima traps, an efficient learning approach for DNMs remains desirable but difficult to implement. In addition to BP, the mainstream DNM-optimization methods include meta-heuristic algorithms (MHAs). However, over the decades, MHAs have developed a large number of different algorithms. How to screen suitable MHAs for optimizing DNMs has become a hot and challenging area of research. In this study, we classify MHAs into different clusters with different population interaction networks (PINs). The performance of DNMs optimized by different clusters of MHAs is tested in the financial time-series-forecasting task. According to the experimental results, the DNM optimized by MHAs with power-law-distributed PINs outperforms the DNM trained based on the BP algorithm.

ResNet and PolyNet Based Identification and (MPC) Control of Dynamical Systems: A Promising Way

This paper deals with model predictive control synthesis which take benefits from artificial neural networks to model (non-linear) dynamical system. More precisely, thanks to a systematic and rigorous methodology, it is shown that residual networks (ResNet) and PolyInception networks (PolyNet) neural network architectures, developed initially for image recognition, are very good candidate for i) identification of dynamical systems, ii) being used as embedded model in a model predictive control laws. Concretely, the widely used non-linear dynamical system quadruple tank process is used as a benchmark. The neural network architectures studied are i) feedforward networks as a reference point, and the two other linked to Euler integration method ii) residual networks and iii) PolyInception networks. Networks training is performed by mixing classical back-propagation algorithm and hyperparameters optimisation through heuristics. The identification results provided show that neural networks of types ii) and iii) perform better than the classical one i), with a better generalisation capability. Finally, model predictive controllers are synthesized based on the various networks trained. The simulation results obtained for controlling water levels of a 4 tanks system benchmark give interesting insights. They show that residual networks based model predictive control is better suited than feedforward networks and PolyInception networks based ones, both taking into account computation time and set point errors.

Using phidelta diagrams to discover relevant patterns in multilayer perceptrons

Understanding the inner behaviour of multilayer perceptrons during and after training is a goal of paramount importance for many researchers worldwide. This article experimentally shows that relevant patterns emerge upon training, which are typically related to the underlying problem difficulty. The occurrence of these patterns is highlighted by means of langle varphi ,delta rangle diagrams, a 2D graphical tool originally devised to support the work of researchers on classifier performance evaluation and on feature assessment. The underlying assumption being that multilayer perceptrons are powerful engines for feature encoding, hidden layers have been inspected as they were in fact hosting new input features. Interestingly, there are problems that appear difficult if dealt with using a single hidden layer, whereas they turn out to be easier upon the addition of further layers. The experimental findings reported in this article give further support to the standpoint according to which implementing neural architectures with multiple layers may help to boost their generalisation ability. A generic training strategy inspired by some relevant recommendations of deep learning has also been devised. A basic implementation of this strategy has been thoroughly used during the experiments aimed at identifying relevant patterns inside multilayer perceptrons. Further experiments performed in a comparative setting have shown that it could be adopted as viable alternative to the classical backpropagation algorithm.

Multidimensional wavelet neural networks Based on polynomial powers of sigmoid

Wavelet functions have been used as the activation function in feed forward neural networks. An abundance of R&D has been produced on wavelet neural network area. Some successful algorithms and applications in wavelet neural network have been developed and reported in the literature. However, most of the aforementioned reports impose many restrictions in the classical back propagation algorithm, such as low dimensionality, tensor product of wavelets, parameters initialization, and, in general, the output is one dimensional, etc. In order to remove some of these restrictions, a family of polynomial wavelets generated from powers of sigmoid functions is presented. We described how a multidimensional wavelet neural networks based on these functions can be constructed, trained and applied in pattern recognition tasks. As examples of applications for the method proposed a framework for face verfication is presented.

A Back Propagation-Type Neural Network Architecture for Solving the Complete n × n Nonlinear Algebraic System of Equations

The objective of this research is the presentation of a neural network capable of solving complete nonlinear algebraic systems of n equations with n unknowns. The proposed neural solver uses the classical back propagation algorithm with the identity function as the output function, and supports the feature of the adaptive learning rate for the neurons of the second hidden layer. The paper presents the fundamental theory associated with this approach as well as a set of experimental results that evaluate the performance and accuracy of the proposed method against other methods found in the literature.

A Computational Framework for Implementation of Neural Networks on Multi-Core Machine

Abstract This paper presents a computational framework, the Generic Programmable Neural Network (GPNN), for efficient implementation of Back-Propagation based neural learning algorithms running on multi-core machines. GPNN has three components: parallelization of neural learning, abstraction of network components, and compile-time generalization. Together these computational components make GPNN an efficient framework for fast implementation of back-propagation based neural learning algorithms, and provide flexibility and reusability for modifying neural network topologies. The GPNN was applied to four different neural learning algorithms: classic back-propagation (BP), quick propagation (QP), resilient propagation (RP) and Levenberg-Marquardt (LM) algorithm. Experiments were conducted to evaluate the effectiveness of GPNN, and results show that the neural learning algorithms implemented in GPNN are more efficient than their respective functions provided by Matlab.

Error-correction learning for artificial neural networks using the Bayesian paradigm. Application to automated medical diagnosis

Automated medical diagnosis models are now ubiquitous, and research for developing new ones is constantly growing. They play an important role in medical decision-making, helping physicians to provide a fast and accurate diagnosis. Due to their adaptive learning and nonlinear mapping properties, the artificial neural networks are widely used to support the human decision capabilities, avoiding variability in practice and errors based on lack of experience. Among the most common learning approaches, one can mention either the classical back-propagation algorithm based on the partial derivatives of the error function with respect to the weights, or the Bayesian learning method based on posterior probability distribution of weights, given training data. This paper proposes a novel training technique gathering together the error-correction learning, the posterior probability distribution of weights given the error function, and the Goodman–Kruskal Gamma rank correlation to assembly them in a Bayesian learning strategy. This study had two main purposes; firstly, to develop anovel learning technique based on both the Bayesian paradigm and the error back-propagation, and secondly,to assess its effectiveness. The proposed model performance is compared with those obtained by traditional machine learning algorithms using real-life breast and lung cancer, diabetes, and heart attack medical databases. Overall, the statistical comparison results indicate that thenovellearning approach outperforms the conventional techniques in almost all respects.

Applied Technology on Artificial Neural Network in Fault Diagnosis System

This paper combined Rumelhart’s adding inertial impulse and dynamically adjusting the learning rate and proposed an improved algorithm to optimize the Back Propagation (BP) networks with applied technology. This improved BP networks is used to determining membership function and applied in fuzzy diagnosing vapor congealing equipment. The application results prove that the improved BP algorithm is effective and the convergence speed is accelerated and is much faster than the classic BP algorithm. The applied technology is very useful in the application course.

Modeling the I-V Curves and Its Derivatives of Microwave Transistors Using Neural Networks

Artificial neural networks are used as a tool in the development of reliable models for field effect transistors. This letter presents an improvement of the classical backpropagation algorithm allowing for the training process information contained in the first derivatives of <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$I_{ds}$</tex></formula> with respect to the control voltages <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$V_{gs}$</tex></formula> and <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$V_{ds}$</tex></formula> . Excellent agreement between measured and simulated data is achieved up to the third order derivative of <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$I_{ds}$</tex> </formula> .

The optimal combination: Grammatical swarm, particle swarm optimization and neural networks

Social behaviour is mainly based on swarm colonies, in which each individual shares its knowledge about the environment with other individuals to get optimal solutions. Such co-operative model differs from competitive models in the way that individuals die and are born by combining information of alive ones. This paper presents the particle swarm optimization with differential evolution algorithm in order to train a neural network instead the classic back propagation algorithm. The performance of a neural network for particular problems is critically dependant on the choice of the processing elements, the net architecture and the learning algorithm. This work is focused in the development of methods for the evolutionary design of artificial neural networks. This paper focuses in optimizing the topology and structure of connectivity for these networks.

A Novel Approach for Pattern Recognition

In this paper, Optical Back Propagation and Levenberg Marquardt (LM) algorithms are used for Pattern Recognition. These two algorithms are compared with Classical Back Propagation algorithm with varying Learning rate and Momentum .The simulation results are obtained and are shown in different graphs. The corresponding simulation results show the efficiency of Levenberg Marquardt (LM) algorithms in comparison to Optical Back Propagation Algorithm and Back Propagation Algorithm.

CBA-neural network control of a non-linear full vehicle model

In this paper, the dynamic behavior of a non-linear eight degrees of freedom vehicle model having active suspensions and passenger seat controlled by a neural network (NN) controller is examined. A robust NN structure is established by using principle design data from the Matlab diagrams of system functions. In the NN structure, Classic Back-Propagation Algorithm (CBA) is employed. The user inputs a set of x 1 − x 16 while the output from the NN consists of f 1 − f 16 non-linear functions. Further, the Permanent Magnet Synchronous Motor (PMSM) controller is also determined using the same NN structure. According to various tests of the NN structure it is demonstrated that the model is able to give highly sensitive outputs for vibration condition, even using a more restricted input data set. The non-linearity occurs due to dry friction on the dampers. The vehicle body and the passenger seat using PMSM are fully controlled at the same time. The time responses of the non-linear vehicle model due to road disturbance and the frequency responses are obtained. Finally, uncontrolled and controlled cases are compared. It is seen that seat vibrations of a non-linear full vehicle model are controlled by NN based system exactly.

Rudder roll stabilization for fishing vessel using neural network approach

This paper presents a neural network (NN) controller for a fishing vessel rudder roll system. The aim of this study is to build a NN controller which uses rudder to regulate both the yaw and roll motion. The neural controller design is accomplished with using the classical back-propagation algorithm (CBA). Effectiveness of the proposed NN control scheme is compared with linear quadratic regulator (LQR) results by simulations carried out a fishing vessel rudder roll stabilizer system.

Neural Optimal Control of PEM Fuel Cells With Parametric CMAC Networks

This paper demonstrates an application of the parametric cerebellar model articulation controller (P-CMAC) network - a neural structure derived from Albus' CMAC algorithm and Takagi-Sugeno-Kang parametric fuzzy inference systems. It resembles the original CMAC proposed by Albus in the sense that it is a local network, i.e., for a given input vector, only a few of the networks neurons will be active and will effectively contribute to the corresponding network output. The internal mapping structure is built in such a way that it implements, for each CMAC memory location, one linear parametric equation of the network input strengths. First, a new approach to design neural optimal control (NOC) systems is proposed. Gradient-descent techniques are still used here to adjust network weights, but this approach has many differences when compared to classical error backpropagation algorithm. Then, P-CMAC is used to control the output voltage of a proton exchange membrane fuel cell (PEM-FC), by means of NOC. The proposed control system allows the definition of an arbitrary performance/cost criterion to be maximized/minimized, resulting in an approximated optimal control strategy. Practical results of PEM-FC voltage behavior at different load conditions are shown, to demonstrate the effectiveness of the NOC algorithm.

Natural gradient learning neural networks for adaptive inversion of Hammerstein systems

This letter applies natural gradient (NG) learning neural networks for adaptive inversion of Hammerstein systems. The system model is composed of a memoryless nonlinearity g(.) followed by a linear filter H. The inverse system is modeled by a neural network composed of an adaptive filter Q followed by a memoryless nonlinear perceptron. The adaptive filter Q aims at inverting the linear part of the system (adaptive deconvolution). The perceptron aims at inverting the memoryless function (adaptive function inversion). The adaptive system is trained using the NG descent algorithm. The letter shows through computer simulations that the NG approach outperforms the classical backpropagation algorithm in terms of mean-squared-error performance and convergence speed.

A HYBRID APPROACH TO KNOWLEDGE ACQUISITION USING NEURAL PETRI NETS AND DS THEORY

Knowledge acquisition from multiple experts and its refinement are important issues in knowledge management of an expert system. The paper presents a novel approach to handling the above problems by combining the synergistic behavior of neural Petri nets and the Dempster–Shafer theory. The Dempster–Shafer theory has been employed here to reduce the scope of uncertainty in the supplied noisy input instances and the inferences generated therefrom by the multiple experts. The noise-free training instances thus obtained are subsequently used to train the neural Petri net model for refining the parameters of its knowledge base. A comparison of the performance of the proposed training algorithm with the classical back-propagation algorithm has also been presented in the paper.

Fault detection and isolation in robotic manipulators via neural networks: A comparison among three architectures for residual analysis

AbstractIn this article we discuss artificial neural networks‐based fault detection and isolation (FDI) applications for robotic manipulators. The artificial neural networks (ANNs) are used for both residual generation and residual analysis. A multilayer perceptron (MLP) is employed to reproduce the dynamics of the robotic manipulator. Its outputs are compared with actual position and velocity measurements, generating the so‐called residual vector. The residuals, when properly analyzed, provides an indication of the status of the robot (normal or faulty operation). Three ANNs architectures are employed in the residual analysis. The first is a radial basis function network (RBFN) which uses the residuals of position and velocity to perform fault identification. The second is again an RBFN, except that it uses only the velocity residuals. The third is an MLP which also performs fault identification utilizing only the velocity residuals. The MLP is trained with the classical back‐propagation algorithm and the RBFN is trained with a Kohonen self‐organizing map (KSOM). We validate the concepts discussed in a thorough simulation study of a Puma 560 and with experimental results with a 3‐joint planar manipulator. © 2001 John Wiley &amp; Sons, Inc.

EEG-based discrimination between imagination of left and right hand movements using adaptive gaussian representation

This article uses the Adaptive Gaussian Representation (AGR) for human electroencephalogram (EEG) feature extraction aiming the discrimination among mental tasks to be used in a brain computer interface (BCI). It does not focus on the AGR time–frequency representation, but rather on their projection coefficients. Ten volunteers were asked to imagine either right or left hand movement, according to a proper visual stimulus. The features of the resulting EEG signals were characterised by extracting AGR coefficients. Classification was carried out using a Multilayer perceptron (MLP) trained with the classical backpropagation algorithm. Overall results show that AGR coefficients representation is able to reveal a significant EEG discrimination between imagination of right and left hand movement with a mean classification performance of 91%±5.8% achieved for female subjects and 87%±5.0% achieved for male subjects.