Conventional Backpropagation Learning Research Articles

Intelligent Financial Forecasting With an Improved Chemical Reaction Optimization Algorithm Based Dendritic Neuron Model

Chemical reaction optimization (CRO) algorithm is a robust metaheuristic by simulating the process of natural chemical reaction and has been applied for solving numerous realistic problems. However, it is suffering from low population diversity, slow convergence, and deprived from local search ability. To boost the search operation of CRO, this article proposes an improved CRO method termed as ICRO. We achieve the improvements in two phases. First, to start the search operation with a better-quality population, we propose a new method to initialize population which helps in enhancing diversification of the search space. Second, to intensify the local search ability we integrate Nelder-Mead simplex method with CRO. The ICRO is then evaluated on a few benchmark functions for optimization and found apposite. Dendritic neuron model (DNM) using additive and multiplicative-based aggregation functions has been emerging as a machine learning approach and found successful in many engineering applications. This study attempts to advance the predictive accuracy of DNM through maintaining a decent steadiness between exploitation and exploration of its search space with the proposed ICRO, termed as ICRO-DNM. The powerful global search ability of ICRO synergies with better approximation capability of DNM thus, able to overcome the limitations of conventional back propagation learning based DNM. ICRO-DNM is evaluated on closing price prediction for four stock index and net asset values of four mutual fund datasets and found efficient. The consequences of this attempt are: (1) Proposed ICRO shows robust parameter optimization ability for benchmark functions and DNM compared to basic CRO, particle swarm optimization (PSO), and genetic algorithm (GA); (2) The learning paradigm formed due to reasonable amalgamation of ICRO and DNM is pretty able to capture the underlying uncertainties coupled with financial data, produces more precise and steady predictions and significantly different from other forecasts.

Fast and improved backpropagation learning of multi‐layer artificial neural network using adaptive activation function

AbstractIn the conventional backpropagation (BP) learning algorithm used for the training of the connecting weights of the artificial neural network (ANN), a fixed slope−based sigmoidal activation function is used. This limitation leads to slower training of the network because only the weights of different layers are adjusted using the conventional BP algorithm. To accelerate the rate of convergence during the training phase of the ANN, in addition to updates of weights, the slope of the sigmoid function associated with artificial neuron can also be adjusted by using a newly developed learning rule. To achieve this objective, in this paper, new BP learning rules for slope adjustment of the activation function associated with the neurons have been derived. The combined rules both for connecting weights and slopes of sigmoid functions are then applied to the ANN structure to achieve faster training. In addition, two benchmark problems: classification and nonlinear system identification are solved using the trained ANN. The results of simulation‐based experiments demonstrate that, in general, the proposed new BP learning rules for slope and weight adjustments of ANN provide superior convergence performance during the training phase as well as improved performance in terms of root mean square error and mean absolute deviation for classification and nonlinear system identification problems.

ResInNet: A Novel Deep Neural Network With Feature Reuse for Internet of Things

Deep neural networks (DNNs) have widely used in various Internet-of-Things (IoT) applications. Pursuing superior performance is always a hot spot in the field of DNN modeling. Recently, feature reuse provides an effective means of achieving favorable nonlinear approximation performance in deep learning. Existing implementations utilizes a multilayer perception (MLP) to act as a functional unit for feature reuse. However, determining connection weight and bias of MLP is a rather intractable problem, since the conventional back-propagation learning approach encounters the limitations of slow convergence and local optimum. To address this issue, this paper develops a novel DNN considering a well-behaved alternative called reservoir computing, i.e., reservoir in network (ResInNet). In this structure, the built-in reservoir has two notable functions. First, it behaves as a bridge between any two restricted Boltzmann machines in the feature learning part of ResInNet, performing a feature abstraction once again. Such reservoir-based feature translation provides excellent starting points for the following nonlinear regression. Second, it serves as a nonlinear approximation, trained by a simple linear regression using the most representative (learned) features. Experimental results over various benchmark datasets show that ResInNet can achieve the superior nonlinear approximation performance in comparison to the baseline models, and produce the excellent dynamic characteristics and memory capacity. Meanwhile, the merits of our approach is further demonstrated in the network traffic prediction related to real-world IoT application.

Novel input and output mapping-sensitive error back propagation learning algorithm for detecting small input feature variations

This paper proposes a new error back propagation learning algorithm that properly enhances the sensitivity of input and output mapping by applying a high-pass filter characteristic to the conventional error back propagation learning algorithm, allowing small input feature variations to be successfully indicated. For the sensitive discrimination of novel class data with slightly different characteristics from the normal class data, the cost function in the proposed neural network algorithm is modified by further increasing the input and output sensitivity, where weight update rules are used to minimize the cost function using a gradient descent method. The proposed algorithm is applied to an auto-associative multilayer perceptron neural network and its performance evaluated with two real-world applications: a laser spot detection-based computer interface system for detecting a laser spot in complex backgrounds and an automatic inspection system for the reliable detection of Mura defects that occur during the manufacture of flat panel liquid crystal displays. When compared with the conventional error back propagation learning algorithm, the proposed algorithm shows a better performance as regards detecting small input feature variations by increasing the input–output mapping sensitivity.

Forecasting River Flow in the USA: A Comparison between Auto-Regression and Neural Network Non-Parametric Models

Forecasting a time series became one of the most challenging tasks to variety of data sets. The existence of large number of parameters to be estimated and the effect of uncertainty and outliers in the measurements makes the time series modeling too complicated. Recently, Artificial Neural Network (ANN) became quite successful tool to handle time series modeling problem. This paper provides a solution to the forecasting problem of the river flow for two well known Rivers in the USA. They are the Black Water River and the Gila River. The selected ANN models were used to train and forecast the daily flows of the first station no: 02047500, for the Black Water River near Dendron in Virginia and the second station no: 0944200 for the Gila River near Clifton in Arizona. The feed forward network is trained using the conventional back propagation learning algorithm with many variations in the NN inputs. We explored models built using various historical data. The selection process of various architectures and training data sets for the proposed NN models are presented. A comparative study of both ANN and the conventional Auto-Regression (AR) model networks indicates that the artificial neural networks performed better than the AR model. Hence, we recommend ANN as a useful tool for river flow forecasting.

An effective learning of neural network by using RFBP learning algorithm

In this paper, an effective learning of neural network by using random fuzzy back-propagation (RFBP) learning algorithm is developed. Based on this new learning algorithm, neural network not only has an accurate learning capability, but also can increase the probability of escaping from the local minimum while neural network is training. For demonstrating the new algorithm we developed has its outperformance, the classifications of the non-convex in two dimensions (NC2) problem are simulated. For comparison, the same simulations by using conventional back-propagation (BP) learning algorithm with constant pairs of learning rate ( α=0.1–0.9) and momentum ( ξ=0.1–0.9) and stochastic BP learning are also performed.

Fast learning algorithms for feedforward neural networks

In order to improve the training speed of multilayer feedforward neural networks (MLFNN), we propose and explore two new fast backpropagation (BP) algorithms obtained: (1) by changing the error functions, in case using the exponent attenuation (or bell impulse) function and the Fourier kernel function as alternative functions; and (2) by introducing the hybrid conjugate-gradient algorithm of global optimization for dynamic learning rate to overcome the conventional BP learning problems of getting stuck into local minima or slow convergence. Our experimental results demonstrate the effectiveness of the modified error functions since the training speed is faster than that of existing fast methods. In addition, our hybrid algorithm has a higher recognition rate than the Polak-Ribieve conjugate gradient and conventional BP algorithms, and has less training time, less complication and stronger robustness than the Fletcher-Reeves conjugate-gradient and conventional BP algorithms for real speech data.

Airborne sonar target recognition using artificial neural network

Airborne sonar target recognition involves two key technical issues: target feature extraction and classification. In this paper, the issue designing a feature classifier with high classification accuracy is discussed. Generally, the multilayered feed-forward neural network can be applied to the airborne sonar target feature classification to achieve the high performance requirement. However, neural networks trained by conventional error back-propagation (B-P) learning algorithms suffer from slow convergence rate and inadequate generalization ability. Detailed analysis of the B-P algorithm reveals that these problems are mainly related to the magnitudes of the components of the gradient vector and the direction of the vector associated with the severely ill-conditioned nature of the Hessian matrix of the error function. A fast back-propagation (F-BP) algorithm is therefore developed to accelerate the learning speed of the B-P algorithm. A dynamic training strategy is then applied to the F-BP algorithm to improve the generalization ability. Experiments are carried out for airborne sonar target feature classification using these algorithms. The results show that the performance of the neural network classifier trained with the proposed algorithm is superior to that of traditional B-P algorithm with a seven-fold learning speed advantage over B-P.

Multigradient: a new neural network learning algorithm for pattern classification

The authors propose a new learning algorithm for multilayer feedforward neural networks, which converges faster and achieves a better classification accuracy than the conventional backpropagation learning algorithm for pattern classification. In the conventional backpropagation learning algorithm, weights are adjusted to reduce the error or cost function that reflects the differences between the computed and the desired outputs. In the proposed learning algorithm, the authors view each term of the output layer as a function of weights and adjust the weights directly so that the output neurons produce the desired outputs. Experiments with remotely sensed data show the proposed algorithm consistently performs better than the conventional backpropagation learning algorithm in terms of classification accuracy and convergence speed.

Backpropagation learning algorithms for classification with fuzzy mean square error

Most of the real life classification problems have ill defined, imprecise or fuzzy class boundaries. Feedforward neural networks with conventional backpropagation learning algorithm are not tailored to this kind of classification problem. Hence, in this paper, feedforward neural networks, that use backpropagation learning algorithm with fuzzy objective functions, are investigated. A learning algorithm is proposed that minimizes an error term, which reflects the fuzzy classification from the point of view of possibilistic approach. Since the proposed algorithm has possibilistic classification ability, it can encompass different backpropagation learning algorithm based on crisp and constrained fuzzy classification. The efficacy of the proposed scheme is demonstrated on a vowel classification problem.

An accelerated learning algorithm for multilayer perceptrons: optimization layer by layer

Multilayer perceptrons are successfully used in an increasing number of nonlinear signal processing applications. The backpropagation learning algorithm, or variations hereof, is the standard method applied to the nonlinear optimization problem of adjusting the weights in the network in order to minimize a given cost function. However, backpropagation as a steepest descent approach is too slow for many applications. In this paper a new learning procedure is presented which is based on a linearization of the nonlinear processing elements and the optimization of the multilayer perceptron layer by layer. In order to limit the introduced linearization error a penalty term is added to the cost function. The new learning algorithm is applied to the problem of nonlinear prediction of chaotic time series. The proposed algorithm yields results in both accuracy and convergence rates which are orders of magnitude superior compared to conventional backpropagation learning.

Acceleration of backpropagation learning using optimised learning rate and momentum

Learning rate and momentum factor are two arbitrary parameters that have to be carefully chosen in the conventional backpropagation (BP) learning algorithm. Based on a linear expansion of the actual outputs of the BP network with respect to the two parameters, the Letter presents an efficient approach to determine the dynamically optimal values of these two parameters. Simulation results indicate that the present approach can provide a remarkable improvement in convergence performance.

Recognition of large-set printed Hangul (Korean script) by two-stage backpropagation neural classifier

A two-stage neural network classifier is described which practically recognizes printed Hangul (Korean script). This classifier is composed of a type classification network and six recognition networks. The former classifies input character images into one of the six types by their overall structure, and then the latter classify them into character code. Furthermore, a training scheme including systematic noises is introduced for improving the generalization capability of the networks. Experiments are conducted with the most frequently used 990 printed Hangul syllables. By the noise included training, the recognition rate amounts to 98.28%, which is better than that of the conventional backpropagation learning. A comparison with a statistical classifier and an analysis of generalization capability confirm the relative superiority of the proposed classification method.

An adaptive least squares algorithm for the efficient training of artificial neural networks

A novel learning algorithm is developed for the training of multilayer feedforward neural networks, based on a modification of the Marquardt-Levenberg least-squares optimization method. The algorithm updates the input weights of each neuron in the network in an effective parallel way. An adaptive distributed selection of the convergence rate parameter is presented, using suitable optimization strategies. The algorithm has better convergence properties than the conventional backpropagation learning technique. Its performance is illustrated, using examples from digital image halftoning and logical operations such as the XOR function.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>