Training Feedforward Networks Research Articles

Analysis on the inherent noise tolerance of feedforward network and one noise-resilient structure

In the past few decades, feedforward neural networks have gained much attraction in their hardware implementations. However, when we realize a neural network in analog circuits, the circuit-based model is sensitive to hardware nonidealities. The nonidealities, such as random offset voltage drifts and thermal noise, may lead to variation in hidden neurons and further affect neural behaviors. This paper considers that time-varying noise exists at the input of hidden neurons, with zero-mean Gaussian distribution. First, we derive lower and upper bounds on the mean square error loss to estimate the inherent noise tolerance of a noise-free trained feedforward network. Then, the lower bound is extended for any non-Gaussian noise cases based on the Gaussian mixture model concept. The upper bound is generalized for any non-zero-mean noise case. As the noise could degrade the neural performance, a new network architecture is designed to suppress the noise effect. This noise-resilient design does not require any training process. We also discuss its limitation and give a closed-form expression to describe the noise tolerance when the limitation is exceeded.

Refinement of global gridded ray-traced Zenith tropospheric delay over Nigeria based on local GNSS network observations

• Application of deep structured supervised neural network modelling technique. • Data derived from the local GNSS observation within the study area. • A refinement model is developed to predict ZTD parameters at user-defined points. • Improvement in the accuracy of the local refine VMF1-ZTD estimates. Recent studies on the application of Zenith Tropospheric Delay (ZTD) in meteorology and geodesy have shown improved accuracy with the use of locally estimated ZTD compared to Global ZTD. This paper focuses on the refinement of global gridded ray-traced Vienna Mapping Functions 1 (VMF1) ZTD over Nigeria, using ZTD estimates derived from local GNSS Continuously Operating Reference Stations (CORS) observations. Three (3) years of GNSS observations collected at 13 CORS located within the study area were post-processed using GNSS Analysis and Processing Software (GAPS) to derive the local GNSS-ZTD estimates. The global VMF1-ZTD were interpolated at the locations of the local GNSS-CORS. An optimal deep structured supervised Neural Network (NN) was derived by training feedforward networks using the Levenberg-Marquardt backpropagation algorithm, with spatiotemporal variables and VMF1-ZTD as inputs and the GNSS-ZTD as the target. The outputs from the NN model were assessed using internal and external data. Results from internal data evaluations indicated a robust and accurate model output with optimum network performances indicating MSE of ∼7.22 × 10 −6 m and ∼4.56 × 10 −4 m for hydrostatic and wet ZTD networks respectively. The external data evaluations performed using the International GNSS Services (IGS) Zenith Path Delay (ZPD), indicated an RMSE range of ∼0.041 m to 0.095 m for global-VMF1 and ∼0.030 m to 0.037 m for the Local Refine (LR)-VMF1. The results suggest that the application of ZTD estimates derived from local GNSS observations in the refinement of global gridded VMF1-ZTD yields an improved accuracy over Nigeria. Consequently, the LR-VMF1 ZTD estimates are recommended for improved accuracy, required for geodetic and meteorological applications in and around Nigeria :

Balanced Gradient Training of Feed Forward Networks

We show that there are infinitely many valid scaled gradients which can be used to train a neural network. A novel training method is proposed that finds the best scaled gradients in each training iteration. The method’s implementation uses first order derivatives which makes it scalable and suitable for deep learning and big data. In simulations, the proposed method has similar or less testing error than conjugate gradient and Levenberg Marquardt. The method reaches the final network utilizing fewer multiplies than the other two algorithms. It also works better than conjugate gradient in convolutional neural networks.

Weight and bias initialization routines for Sigmoidal Feedforward Network

The success of the Sigmoidal Feedforward Networks in the solution of complex learning task can be attributed to their Universal Approximation Property. These networks are trained using non-linear iterative optimization method (of first-order or second-order) to solve a learning task. The convergence rate in Sigmoidal Feedforward Network training is affected by the initial choice of weights, therefore, in this paper, we propose two new weight initialization routines (Routine-1 and Routine-2) using characteristics of input and output data and property of activation function. Routine-1 uses the linear dependency of weight update step size on derivative of activation function and thus, initialize weights and bias to activate the activation function region near zero (input), where the derivative is maximum, therefore, increasing the weight update step size, and hence, the convergence speed. The same principle is used to derive Routine-2, that initialize weights and bias to activate distinct point in the significant range of activation function (where significant range defines the non-saturated region in activation function), such that, each node evolves independently of each other, and act as distinct feature identifier. Initializing weights in significant range reduces chances of (hidden) nodes getting stuck in saturated state. The networks initialized using proposed routines has higher convergence and higher probability to achieve deeper minima. The efficiency of proposed routines is evaluated by comparing them to conventional random weight initialization routine and 11 weight initialization routines proposed in literature (4 well established routines and 7 recently proposed routines) for several benchmark problems. The proposed routine is also tested for larger networks sizes and larger datasets such as MNIST. The results show that the performance of proposed routines is better than conventional random weight initialization routine and 11 established weight initialization routines.

Bayesian regularized artificial neural networks for the estimation of the probability of default

Artificial neural networks (ANNs) have been extensively used for classification problems in many areas such as gene, text and image recognition. Although ANNs are popular also to estimate the probability of default in credit risk, they have drawbacks; a major one is their tendency to overfit the data. Here we propose an improved Bayesian regularization approach to train ANNs and compare it to the classical regularization that relies on the back-propagation algorithm for training feed-forward networks. We investigate different network architectures and test the classification accuracy on three data sets. Profitability, leverage and liquidity emerge as important financial default driver categories.

Fast Continuous Structural Similarity Patch Based Arbitrary Style Transfer

Style transfer is using a pair of content and style images to synthesize a stylized image which has both the structure of the content image and the style of style image. Existing optimization-based methods are limited in their performance. Some works using a feed-forward network allow arbitrary style transfer but cannot reflect the style. In this paper, we present a fast continuous structural similarity patch based arbitrary style transfer. Firstly, we introduce the structural similarity index (SSIM) to compute the similarity between all of the content and style patches for obtaining their similarity. Then a local style patch choosing procedure is applied to maximize the utilization of all style patches and make the swapped style patch continuous matching with respect to the spatial location of style at the same time. Finally, we apply an efficient trained feed-forward inverse network to obtain the final stylized image. We use more than 80,000 natural images and 120,000 style images to train that feed-forward inverse network. The results show that our method is able to transfer arbitrary style with consistency, and the result comparison stage is made to show the effectiveness and high-quality of our stylized images.

“Parallel Training Considered Harmful?”: Comparing series-parallel and parallel feedforward network training

Neural network models for dynamic systems can be trained either in parallel or in series-parallel configurations. Influenced by early arguments, several papers justify the choice of series-parallel rather than parallel configuration claiming it has a lower computational cost, better stability properties during training and provides more accurate results. Other published results, on the other hand, defend parallel training as being more robust and capable of yielding more accu- rate long-term predictions. The main contribution of this paper is to present a study comparing both methods under the same unified framework. We focus on three aspects: i) robustness of the estimation in the presence of noise; ii) computational cost; and, iii) convergence. A unifying mathematical framework and simulation studies show situations where each training method provides better validation results, being parallel training better in what is believed to be more realistic scenarios. An example using measured data seems to reinforce such claim. We also show, with a novel complexity analysis and numerical examples, that both methods have similar computational cost, being series series-parallel training, however, more amenable to parallelization. Some informal discussion about stability and convergence properties is presented and explored in the examples.

A simple feedforward convolutional conceptor neural network for classification

Training deep neural networks usually invokes error gradient backpropagation (BP), which costs a long training time even on high-performance computing devices or with professional skills of experienced experimenters. In order to accelerate and simplify the training process of deep networks, non-iterative training procedures have been proposed. Here we present another non-iterative approach, Feedforward Convolutional Conceptor Neural Network (FCCNN), for training feedforward networks on image classification tasks. Our work makes two major contributions: (1) a conceptor based classifier which is specific for non-temporal data; (2) a simple non-iterative neural network model. The architecture is established based on the combination of a Convolutional Neural Network (CNN) shaped by an unsupervised feature learning based on Principal Component Analysis (PCA), binary thresholding and the proposed non-temporal conceptor classifier. Through various experiments on MNIST variation datasets, FCCNN achieves classifying accuracy comparable to the state-of-the-art methods while requires significantly reduced training time.

An Improved ACOBP Algorithm for Load Forecasting in Power System

In this paper, a new ant colony algorithm is introduced and a new feedforward network training strategy, ACOBP algorithm, is proposed for the first time, based on the traditional linear feedforward neural network and BP algorithm. In this paper, the network is used for load forecasting and the results of the example show that this method is feasible and reasonable. In the aspect of sample data processing, this paper has fully considered the influence of several main factors on the load variation, and quantified a series of weather factors, such as temperature and rainfall, and forecasted the important holiday separately. By analyzing the special law of load, the network parameters are adjusted and the precision of prediction is improved.

Automated identification of copepods using digital image processing and artificial neural network

BackgroundCopepods are planktonic organisms that play a major role in the marine food chain. Studying the community structure and abundance of copepods in relation to the environment is essential to evaluate their contribution to mangrove trophodynamics and coastal fisheries. The routine identification of copepods can be very technical, requiring taxonomic expertise, experience and much effort which can be very time-consuming. Hence, there is an urgent need to introduce novel methods and approaches to automate identification and classification of copepod specimens. This study aims to apply digital image processing and machine learning methods to build an automated identification and classification technique.ResultsWe developed an automated technique to extract morphological features of copepods' specimen from captured images using digital image processing techniques. An Artificial Neural Network (ANN) was used to classify the copepod specimens from species Acartia spinicauda, Bestiolina similis, Oithona aruensis, Oithona dissimilis, Oithona simplex, Parvocalanus crassirostris, Tortanus barbatus and Tortanus forcipatus based on the extracted features. 60% of the dataset was used for a two-layer feed-forward network training and the remaining 40% was used as testing dataset for system evaluation. Our approach demonstrated an overall classification accuracy of 93.13% (100% for A. spinicauda, B. similis and O. aruensis, 95% for T. barbatus, 90% for O. dissimilis and P. crassirostris, 85% for O. similis and T. forcipatus).ConclusionsThe methods presented in this study enable fast classification of copepods to the species level. Future studies should include more classes in the model, improving the selection of features, and reducing the time to capture the copepod images.

Memetic cooperative coevolution of Elman recurrent neural networks

Cooperative coevolution decomposes an optimisation problem into subcomponents and collectively solves them using evolutionary algorithms. Memetic algorithms provides enhancement to evolutionary algorithms with local search. Recently, the incorporation of local search into a memetic cooperative coevolution method has shown to be efficient for training feedforward networks on pattern classification problems. This paper applies the memetic cooperative coevolution method for training recurrent neural networks on grammatical inference problems. The results show that the proposed method achieves better performance in terms of optimisation time and robustness.

An integrated growing-pruning method for feedforward network training

In order to facilitate complexity optimization in feedforward networks, several algorithms are developed that combine growing and pruning. First, a growing scheme is presented which iteratively adds new hidden units to full-trained networks. Then, a non-heuristic one-pass pruning technique is presented, which utilizes orthogonal least squares. Based upon pruning, a one-pass approach is developed for generating the validation error versus network size curve. A combined approach is described in which networks are continually pruned during the growing process. As a result, the hidden units are ordered according to their usefulness, and the least useful units are eliminated. Examples show that networks designed using the combined method have less training and validation error than growing or pruning alone. The combined method exhibits reduced sensitivity to the initial weights and generates an almost monotonic error versus network size curve. It is shown to perform better than two well-known growing methods—constructive backpropagation and cascade correlation.

A new EM-based training algorithm for RBF networks

In this paper, we propose a new Expectation–Maximization (EM) algorithm which speeds up the training of feedforward networks with local activation functions such as the Radial Basis Function (RBF) network. In previously proposed approaches, at each E-step the residual is decomposed equally among the units or proportionally to the weights of the output layer. However, these approaches tend to slow down the training of networks with local activation units. To overcome this drawback in this paper we use a new E-step which applies a soft decomposition of the residual among the units. In particular, the decoupling variables are estimated as the posterior probability of a component given an input–output pattern. This adaptive decomposition takes into account the local nature of the activation function and, by allowing the RBF units to focus on different subregions of the input space, the convergence is improved. The proposed EM training algorithm has been applied to the nonlinear modeling of a MESFET transistor.

Two highly efficient second-order algorithms for training feedforward networks

We present two highly efficient second-order algorithms for the training of multilayer feedforward neural networks. The algorithms are based on iterations of the form employed in the Levenberg-Marquardt (LM) method for nonlinear least squares problems with the inclusion of an additional adaptive momentum term arising from the formulation of the training task as a constrained optimization problem. Their implementation requires minimal additional computations compared to a standard LM iteration. Simulations of large scale classical neural-network benchmarks are presented which reveal the power of the two methods to obtain solutions in difficult problems, whereas other standard second-order techniques (including LM) fail to converge.

Training feedforward networks using simultaneous perturbation with dynamic tunneling

An efficient technique, namely simultaneous perturbation with dynamic tunneling for training single hidden layer feedforward network, is proposed. A sigmoidal hidden neuron is added to the single hidden layer neural network after training. Then the cascaded network is trained again using simultaneous perturbation. The dynamic tunneling technique is employed to detrap the local minima in training. The proposed technique is shown to give better convergence results for the selected problems, namely neuro-controller, encoder, adder, demultiplexer, XOR and L–T character recognition problem.

Feedforward networks training speed enhancement by optimal initialization of the synaptic coefficients

This letter aims at determining the optimal bias and magnitude of initial weight vectors based on multidimensional geometry. This method ensures the outputs of neurons are in the active region and the range of the activation function is fully utilized. In this letter, very thorough simulations and comparative study were performed to validate the performance of the proposed method. The obtained results on five well-known benchmark problems demonstrate that the proposed method deliver consistent good results compared with other weight initialization methods.

Adaptive training and pruning in feedforward networks

A local extended Kalman filter training and pruning approach is proposed to train feedforward networks with the goal of reducing the computational complexity and storage requirement in large-scale practical problems. The performance of the proposed algorithm is demonstrated for the problem of handwritten digit recognition.

Modifying Weights Layer-By-Layer with Levenberg-Marquardt Backpropagation Algorithm

Abstract Training feedforward networks, layer–by–layer, with the Levenberg–Marquardt back–propagation algorithm is presented in this paper. The Levenberg–Marquardt backpropagation technique has been noted as an efficient method for training feedforward neural networks in terms of training accuracy, convergence properties and overall training time. We introduce a method to further improve the computation and memory complexity of this algorithm by modifying the weights layer–by–layer. Four examples from the literature and from an engineering application are provided to demonstrate the outperformance of the technique over the general Levenberg–Marquardt backpropagation, which is based on adjusting all the weights simultaneously. These examples show that further improvement, in both the training time and convergence property, can be obtained using the new approach.

Extended least squares based algorithm for training feedforward networks

An extended least squares-based algorithm for feedforward networks is proposed. The weights connecting the last hidden and output layers are first evaluated by least squares algorithm. The weights between input and hidden layers are then evaluated using the modified gradient descent algorithms. This arrangement eliminates the stalling problem experienced by the pure least squares type algorithms; however, still maintains the characteristic of fast convergence. In the investigated problems, the total number of FLOPS required for the networks to converge using the proposed training algorithm are only 0.221%-16.0% of that using the Levenberg-Marquardt algorithm. The number of floating point operations per iteration of the proposed algorithm are only 1.517-3.521 times of that of the standard backpropagation algorithm.

Bootstrapping with Noise: An Effective Regularization Technique

Bootstrap samples with noise are shown to be an effective smoothness and capacity control technique for training feedforward networks and for other statistical methods such as generalized additive models. It is shown that noisy bootstrap performs best in conjunction with weight-decay regularization and ensemble averaging. The two-spiral problem, a highly non-linear, noise-free data, is used to demonstrate these findings. The combination of noisy bootstrap and ensemble averaging is also shown useful for generalized additive modelling, and is also demonstrated on the well-known Cleveland heart data.