Training Feedforward Neural Networks Research Articles

A New Quantum Tunneling Particle Swarm Optimization Algorithm for Training Feedforward Neural Networks

A New Quantum Tunneling Particle Swarm Optimization Algorithm for Training Feedforward Neural Networks

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Based on a novel algorithm, known as the upper-layer-solution-aware (USA), a new algorithm, in which the penalty method is introduced into the empirical risk, is studied for training feed-forward neural networks in this paper, named as USA with penalty. Both theoretical analysis and numerical results show that it can control the magnitude of weights of the networks. Moreover, the deterministic theoretical analysis of the new algorithm is proved. The monotonicity of the empirical risk with penalty term is guaranteed in the training procedure. The weak and strong convergence results indicate that the gradient of the total error function with respect to weights tends to zero, and the weight sequence goes to a fixed point when the iterations approach positive infinity. Numerical experiment has been implemented and effectively verifies the proved theoretical results.

Feed-forward neural network training using sparse representation

The feed-forward neural network (FNN) has drawn great interest in many applications due to its universal approximation capability. In this paper, a novel algorithm for training FNNs is proposed using the concept of sparse representation. The major advantage of the proposed algorithm is that it is capable of training the initial network and optimizing the network structure simultaneously. The proposed algorithm consists of two core stages: structure optimization and weight update. In the structure optimization stage, the sparse representation technique is employed to select important hidden neurons that minimize the residual output error. In the weight update stage, a dictionary learning based method is implemented to update network weights by maximizing the output diversity from hidden neurons. This weight-updating process is designed to improve the performance of the structure optimization. Based on several benchmark classification and regression problems, we present experimental results comparing the proposed algorithm with state-of-the-art methods. Simulation results show that the proposed algorithm offers comparative performance in terms of the final network size and generalization ability.

Global optimization issues in deep network regression: an overview

The paper presents an overview of global issues in optimization methods for training feedforward neural networks (FNN) in a regression setting. We first recall the learning optimization paradigm for FNN and we briefly discuss global scheme for the joint choice of the network topologies and of the network parameters. The main part of the paper focuses on the core subproblem which is the continuous unconstrained (regularized) weights optimization problem with the aim of reviewing global methods specifically arising both in multi layer perceptron/deep networks and in radial basis networks. We review some recent results on the existence of non-global stationary points of the unconstrained nonlinear problem and the role of determining a global solution in a supervised learning paradigm. Local algorithms that are widespread used to solve the continuous unconstrained problems are addressed with focus on possible improvements to exploit the global properties. Hybrid global methods specifically devised for FNN training optimization problems which embed local algorithms are discussed too.

Levenberg–Marquardt Backpropagation Training of Multilayer Neural Networks for State Estimation of a Safety-Critical Cyber-Physical System

As an important safety-critical cyber-physical system (CPS), the braking system is essential to the safe operation of the electric vehicle. Accurate estimation of the brake pressure is of great importance for automotive CPS design and control. In this paper, a novel probabilistic estimation method of brake pressure is developed for electrified vehicles based on multilayer artificial neural networks (ANNs) with Levenberg–Marquardt backpropagation (LMBP) training algorithm. First, the high-level architecture of the proposed multilayer ANN for brake pressure estimation is illustrated. Then, the standard backpropagation (BP) algorithm used for training of the feed-forward neural network (FFNN) is introduced. Based on the basic concept of BP, a more efficient training algorithm of LMBP method is proposed. Next, real vehicle testing is carried out on a chassis dynamometer under standard driving cycles. Experimental data of the vehicle and the powertrain systems are collected, and feature vectors for FFNN training collection are selected. Finally, the developed multilayer ANN is trained using the measured vehicle data, and the performance of the brake pressure estimation is evaluated and compared with other available learning methods. Experimental results validate the feasibility and accuracy of the proposed ANN-based method for braking pressure estimation under real deceleration scenarios.

Performance comparison of feedforward neural network training algorithms in modeling for synthesis of polycaprolactone via biopolymerization

This paper reports the biopolymerization of e-caprolactone, using lipase Novozyme 435 catalyst at varied impeller speeds and reactor temperatures. A multilayer feedforward neural network (FFNN) model with 11 different training algorithms is developed for the multivariable nonlinear biopolymerization of polycaprolactone (PCL). In previous works, biopolymerization carried out in scaled-up bioreactors is modeled through FFNN. No review discussed the role of different training algorithms in artificial neural network on the estimation of biopolymerization performance. This paper compares mean absolute error, mean square error, and mean absolute percentage error (MAPE) in the PCL biopolymerization process for 11 different training algorithms that belong to six classes, namely (1) additive momentum, (2) self-adaptive learning rate, (3) resilient backpropagation, (4) conjugate gradient backpropagation, (5) quasi-Newton, and (6) Bayesian regulation propagation. This paper aims to identify the most effective training method for biopolymerization. Results show that the quasi-Newton-based and Levenberg–Marquardt algorithms have the best performance with MAPE values of 4.512, 5.31, and 3.21% for the number of average molecular weight, weight average molecular weight, and polydispersity index, respectively.

A Novel Multimean Particle Swarm Optimization Algorithm for Nonlinear Continuous Optimization: Application to Feed-Forward Neural Network Training

Multilayer feed-forward artificial neural networks are one of the most frequently used data mining methods for classification, recognition, and prediction problems. The classification accuracy of a multilayer feed-forward artificial neural networks is proportional to training. A well-trained multilayer feed-forward artificial neural networks can predict the class value of an unseen sample correctly if provided with the optimum weights. Determining the optimum weights is a nonlinear continuous optimization problem that can be solved with metaheuristic algorithms. In this paper, we propose a novel multimean particle swarm optimization algorithm for multilayer feed-forward artificial neural networks training. The proposed multimean particle swarm optimization algorithm searches the solution space more efficiently with multiple swarms and finds better solutions than particle swarm optimization. To evaluate the performance of the proposed multimean particle swarm optimization algorithm, experiments are conducted on ten benchmark datasets from the UCI repository and the obtained results are compared to the results of particle swarm optimization and other previous research in the literature. The analysis of the results demonstrated that the proposed multimean particle swarm optimization algorithm performed well and it can be adopted as a novel algorithm for multilayer feed-forward artificial neural networks training.

Training Feedforward Neural Networks Using Social Learning Particle Swarm Optimization- A Case Comparison Study on Electrical System

The knowledge about training feed forward neural networks (FNNs) is an important and complex issue in the supervised learning field. In the process of learning, the FNNs system involves some input parameters such as connection weights and biases, which may greatly influence the performance of FNNs training. In this paper, a newly developed meta-heuristic method, named social learning particle swarm optimization (SLPSO), is trying to find the optimal combination of connection weights and biases for FNNs, which is often used to deal with power load forecasting problem. In the numerical experiments, a case on the power load forecasting problem is employed to verify the effectiveness of SLPSO. The experiment results indicate that SLPSO has the advantages on the training accuracy and testing accuracy with respect to other six state-of-the-art intelligent optimization algorithms.

Dynamic group optimisation algorithm for training feed-forward neural networks

Feed-forward neural networks are efficient at solving various types of problems. However, finding efficient training algorithms for feed-forward neural networks is challenging. The dynamic group op ...

Neural-network-based order parameters for classification of binary hard-sphere crystal structures

ABSTRACTIdentifying crystalline structures is a common challenge in many types of research. Here, we focus on binary mixtures of hard spheres of various size ratios, which stabilise a range of crystal structures with varying complexity. We train feed-forward neural networks to distinguish different crystalline and fluid environments on a single-particle basis, by analysing vectors composed of several averaged local bond order parameters. For all size ratios considered, we achieve a classification accuracy above for all phases, meaning that our method is completely general and able to capture structural differences of a wide range of binary crystals.

Non-iterative approaches in training feed-forward neural networks and their applications

Focusing on non-iterative approaches in training feed-forward neural networks, this special issue includes 12 papers to share the latest progress, current challenges, and potential applications of this topic. This editorial presents a background of the special issue and a brief introduction to the 12 contributions.

Application of Islanding Detection and Classification of Power Quality Disturbance in Hybrid Energy System

Islanding and power quality (PQ) disturbances in hybrid energy system become more serious with the application of renewable energy sources. In this paper, a novel method based on wavelet transform (WT) and modified feed forward neural network (FNN) is proposed to detect islanding and classify PQ problems. First, the performance indices, i.e., the energy content and SD of the transformed signal are extracted from the negative sequence component of the voltage signal at PCC using WT. Afterward, WT indices are fed to train FNNs midfield by Particle Swarm Optimization (PSO) which is a novel heuristic optimization method. Then, the results of simulation based on WT-PSOFNN are discussed in MATLAB/SIMULINK. Simulations on the hybrid power system show that the accuracy can be significantly improved by the proposed method in detecting and classifying of different disturbances connected to multiple distributed generations.

Constructive Neural Network Learning.

In this paper, we aim at developing scalable neural network-type learning systems. Motivated by the idea of constructive neural networks in approximation theory, we focus on constructing rather than training feed-forward neural networks (FNNs) for learning, and propose a novel FNNs learning system called the constructive FNN (CFN). Theoretically, we prove that the proposed method not only overcomes the classical saturation problem for constructive FNN approximation, but also reaches the optimal learning rate when the regression function is smooth, while the state-of-the-art learning rates established for traditional FNNs are only near optimal (up to a logarithmic factor). A series of numerical simulations are provided to show the efficiency and feasibility of CFN.

Flower pollination–feedforward neural network for load flow forecasting in smart distribution grid

Nature-inspired population-based metaheuristic flower pollination algorithm is proposed in solving load flow forecasting problem in smart distribution grid environment. The efficient approach involves training a feedforward neural network (FNN) with a new flower pollination algorithm (FPA). The idea is to perform short-term load flow forecasting in smart distribution network, thus maintaining system security due to intermittency of renewable energy penetration and power flow demand. Application of optimization algorithms such as FPA in training neural network improves accuracy, overcomes generalization ability of neural network, requires less data and prevents premature convergence problem in artificial intelligence solutions due to nonlinearity of parameters. The real load flow data are collected through distribution management system of Konya Organized Industrial Zone. The result obtained indicates strong improvement in error reduction using flower pollination optimization algorithm in training FNN for short-term load flow forecasting in smart distribution grid; the model is compared against FNN model and efficient support vector regression.

FORECASTING LONG TERM PRECIPITATION USING CUCKOO SEARCH OPTIMIZATION NEURAL NETWORK MODELS

With anticipation of global warming and climate change, quantitative prediction of future precipitation trend is more important than ever. Global circulation models (GCMs) are widely used as the base for simulating climate change. However, due to their coarse resolution, researchers have been using various downscaling techniques to produce finer model for regional use. Recent advancements in metaheuristic algorithms have provided an alternative approach in downscaling. This paper introduces the application of a novel optimization algorithm, named as Cuckoo Search Optimization (CSO), to train Feedforward and Recurrent neural network to forecast long term precipitation. As benchmark, CSO was compared with Scaled Conjugate Gradient (SCG) and Levenberg-Marquardt (LM) methods. The models were evaluated through validation with historical precipitation; as well as their performance in Pearson correlation (r), root mean square error (RMSE), mean absolute error (MAE), and mean bias (MB). Results showed that CSO is capable of forecasting precipitation up to 90%~100% confidence level with an overall lower mean absolute error, root mean square error and mean bias; outperforming SCG and LM. Future precipitation forecasts revealed that the city will experience an increase of mean annual precipitation by 6~7% over Year 2071-2100. A regional climate model (RCM) with finer resolution was also investigated. Preliminary results revealed an underperformance of the regional climate model due to weaker correlation link between the predictors and historical precipitation.

An effective gbest-guided gravitational search algorithm for real-parameter optimization and its application in training of feedforward neural networks

Gravitational search algorithm (GSA) is a recently introduced meta-heuristic that has shown great performance in numerical function optimization and solving real world problems. GSA provides an excellent social interaction between its search agents. This social interaction results in admirable exploration of the search space and gives a unique social component to GSA. However, the social interaction is not able to exploit good solutions in an efficient manner. To overcome this problem, a novel algorithm named as gbest-guided gravitational search algorithm (GG-GSA) has been proposed by utilizing the global best (gbest) solution in the force calculation equation of GSA. The employment of gbest solution in any optimization algorithm is a tough task and can lead to premature convergence. In the proposed algorithm, the gbest solution is used adaptively and is able to achieve a better trade-off between exploration and exploitation. The performance of the proposed algorithm is compared with GSA and its variants on different suites of well-known benchmark test functions. The experimental results show that the GG-GSA performs better than other algorithms for most of the benchmark test functions. Furthermore, to test the ability of the proposed algorithm in solving real world applications, training of feedforward neural network problem is chosen. The results demonstrated the exceptional performance of GG-GSA on real world data-set.

Feed-Forward Neural Network-Based Predictive Image Coding for Medical Image Compression

The generation of high volume of medical images in recent years has increased the demand for more efficient compression methods to cope up with the storage and transmission problems. In the case of medical images, it is important to ensure that the compression process does not affect the image quality adversely. In this paper, a predictive image coding method is proposed which preserves the quality of the medical image in the diagnostically important region (DIR) even after compression. In this method, the image is initially segmented into two portions, namely DIR and non-DIR portions, using a graph-based segmentation procedure. The prediction process is implemented using two identical feed-forward neural networks (FF-NNs) at the compression and decompression stages. Gravitational search and particle swarm algorithms are used for training the FF-NNs. Prediction is performed in both a lossless (LLP) and near-lossless (NLLP) manner for evaluating the performances of the two FF-NN training algorithms. The prediction error sequence which is the difference between the actual and predicted pixel values is further compressed using a Markov model-based arithmetic coding. The proposed method is tested using CLEF med 2009 database. The experimental results demonstrate that the proposed method is equipped for compressing the medical images with minimum degradation in the image quality. It is found that the gravitational search method achieves higher PSNR values compared to the particle swarm and backpropagation methods.

A novel conjugate gradient method with generalized Armijo search for efficient training of feedforward neural networks

Abstract In this paper, a novel multilayer backpropagation (BP) neural network model is proposed based on conjugate gradient (CG) method with generalized Armijo search. The presented algorithm requires low memory and performs fast convergent speed in practical applications. One reason is that the constructed conjugate direction guarantees the sufficient descent behavior in minimizing the given objective function. The other stems from the fact that the generalized Armijo method can automatically determine a more suitable learning rate in each training epoch. As a theoretical contribution, two deterministic convergent results, weak and strong convergence, have been detailedly proved under more relaxed assumptions. The weak convergence means that the norm of gradient of the objective function tends to zero. For the strong convergence, it represents that the sequence of weight vectors approaches a fixed point. To support the theoretical results, some illustrated simulations have been done on various benchmark datasets.

A review on neural networks with random weights

In big data fields, with increasing computing capability, artificial neural networks have shown great strength in solving data classification and regression problems. The traditional training of neural networks depends generally on the error back propagation method to iteratively tune all the parameters. When the number of hidden layers increases, this kind of training has many problems such as slow convergence, time consuming, and local minima. To avoid these problems, neural networks with random weights (NNRW) are proposed in which the weights between the hidden layer and input layer are randomly selected and the weights between the output layer and hidden layer are obtained analytically. Researchers have shown that NNRW has much lower training complexity in comparison with the traditional training of feed-forward neural networks. This paper objectively reviews the advantages and disadvantages of NNRW model, tries to reveal the essence of NNRW, gives our comments and remarks on NNRW, and provides some useful guidelines for users to choose a mechanism to train a feed-forward neural network.

Back projection: An effective postprocessing method for GAN-based face sketch synthesis

We consider the image transformation problems in this paper, where an input face photo is transformed into a sketch, i.e. face sketch synthesis. It plays important role in video surveillance-based law enforcement. Recent methods for such problems typically train feed-forward convolutional neural networks (CNN) or graphical probabilistic models. In this paper, inspired by the recent success in generating images of generative adversarial networks (GAN), we employ GAN to perform this task. However, accompanying with fine textures generated by GAN model, noise appears among the generated results. We proposed a back projection method to reconstruct the synthesized results. Extensive experiments on public face databases illustrate the effectiveness and superiority of the proposed method compared with state-of-the-art methods. The proposed back projection strategy can be extended to other GAN-based image-to-image translation problems. Data and implementation code in this paper are available online at www.ihitworld.com/WNN/Back_Projection.zip.