Training Feedforward Neural Networks Research Articles

Enhanced Salp Swarm Algorithm based on random walk and its application to training feedforward neural networks

Salp Swarm Algorithm (SSA) is a new type of metaheuristic and has shown superiority over other well-known algorithms such as Particle Swarm Optimization and Grey Wolf Optimizer in solving challenging optimization problems. Despite its superior performance, SSA still has problems such as insufficient convergence speed. Moreover, its local optima avoidance ability is not as good as those evolutionary algorithms using crossover operators. In this paper, we propose a modified Salp Swarm Algorithm (m-SSA) which improves the exploitation and exploration of SSA by integrating random walk strategy and especially enhances exploration by adding a new controlling parameter. In addition, a simulated annealing-type acceptance criterion is adopted to accept the fittest follower position as the new best leader position. The performance of the proposed algorithm is benchmarked on a set of classical functions and CEC2014 test suite. The proposed algorithm (m-SSA) outperforms SSA significantly on most test functions. When compared with other state-of-the-art metaheuristics, it also presents very competitive results. Besides, we apply the proposed algorithm on training feedforward neural networks (FNNs) and the results prove the effectiveness and efficiency of m-SSA.

Multi-Start Method with Cutting for Solving Problems of Unconditional Optimization

In the present paper, we present a multi-start method with dynamic cutting of “unpromising” starts of locally optimal algorithms for solving continuous unconditional optimization problems. We also describe our results of testing of the proposed method for solving problems of feed-forward neural network training.

Wall-modeled large-eddy simulations of spanwise rotating turbulent channels—Comparing a physics-based approach and a data-based approach

We develop wall modeling capabilities for large-eddy simulations (LESs) of channel flow subjected to spanwise rotation. The developed models are used for flows at various Reynolds numbers and rotation numbers, with different grid resolutions and in differently sized computational domains. We compare a physics-based approach and a data-based machine learning approach. When pursuing a data-based approach, we use the available direct numerical simulation data as our training data. We highlight the difference between LES wall modeling, where one writes all flow quantities in a coordinate defined by the wall-normal direction and the near-wall flow direction, and Reynolds-averaged Navier-Stokes modeling, where one writes flow quantities in tensor forms. Pursuing a physics-based approach, we account for system rotation by reformulating the eddy viscosity in the wall model. Employing the reformulated eddy viscosity, the wall model is able to predict the mean flow correctly. Pursuing a data-based approach, we train a fully connected feed-forward neural network (FNN). The FNN is informed about our knowledge (although limited) on the mean flow. We then use the trained FNNs as wall models in wall modeled LES (WMLES) and show that it predicts the mean flow correctly. While it is not the focus of this study, special attention is paid to the problem of log-layer mismatch, which is common in WMLES. Our study shows that log-layer mismatch, or rather, linear-layer mismatch in WMLES of spanwise rotating channels, is not present at high rotation numbers, even when the wall-model/LES matching location is at the first grid point.

Performance comparison of feedforward neural network training algorithms in modelling microwave pyrolysis of oil palm fibre for hydrogen and biochar production

AbstractMicrowave pyrolysis of oil palm fibre (OPF) was conducted to study the effect of microwave power, temperature, and nitrogen (N2) flow rate on the production of hydrogen and biochar. The effect of microwave power ranging from 400 to 900 W, reaction temperatures ranging from 450°C to 700°C, and N2 flow rates ranging 200 to 1,200 cm3 min−1 were investigated. The microwave power, reaction temperature, and N2 flow rates are directly proportional to hydrogen and inversely proportional to biochar yield. The characteristics of the biochar were analysed using CHNOS, scanning electron microscopy, and Brunauer–Emmett–Teller results. Central composite design was used to optimize the number of experiment. The optimized experimental datasets were used as inputs for developing a model that can predict the weight of hydrogen and biochar from microwave pyrolysis of OPF. A multilayer feedforward neural network model was developed by comparing the performance of 11 different types of backpropagation training algorithms. The 11 training algorithms belong to six classes, which are (a) additive momentum, (b) self‐adaptive learning rate, (c) resilient backpropagation, (d) conjugate gradient, (e) quasi‐Newton, and (f) Bayesian regulation (BR). Best performing training algorithm was selected based on the lowest error values computed. The objective of this research is to identify the most suitable training algorithm for this process. Levenberg–Marquardt and BR exhibited very good performance. BR exhibited best performance with lowest error values in predicting weight of hydrogen and biochar. The testing data error values for hydrogen weight is 0.216 root mean square error (RMSE), 0.144 mean absolute error (MAE), and 0.020 mean absolute percentage error (MAPE), and biochar is 0.886 RMSE, 0.489 MAE, and 0.019 MAPE. The optimum number of hidden neuron for this algorithm is 19 with mean square error 0.23. The biochar exhibits porous structure suggesting possible land application.

Machine Learning, Urban Water Resources Management and Operating Policy

Meticulously analyzing all contemporaneous conditions and available options before taking operations decisions regarding the management of the urban water resources is a necessary step owing to water scarcity. More often than not, this analysis is challenging because of the uncertainty regarding inflows to the system. The most common approach to account for this uncertainty is to combine the Bayesian decision theory with the dynamic programming optimization method. However, dynamic programming is plagued by the curse of dimensionality, that is, the complexity of the method is proportional to the number of discretized possible system states raised to the power of the number of reservoirs. Furthermore, classical statistics does not consistently represent the stochastic structure of the inflows (see persistence). To avoid these problems, this study will employ an appropriate stochastic model to produce synthetic time-series with long-term persistence, optimize the system employing a network flow programming modelling, and use the optimization results for training a feedforward neural network (FFN). This trained FFN alone can serve as a decision support tool that describes not only reservoir releases but also how to operate the entire water supply system. This methodology is applied in a simplified representation of the Athens water supply system, and the results suggest that the FFN is capable of successfully operating the system according to a predefined operating policy.

Random Regrouping and Factorization in Cooperative Particle Swarm Optimization Based Large-Scale Neural Network Training

Previous studies have shown that factorization and random regrouping significantly improve the performance of the cooperative particle swarm optimization (CPSO) algorithm. However, few studies have examined whether this trend continues when CPSO is applied to the training of feed forward neural networks. Neural network training problems often have very high dimensionality and introduce the issue of saturation, which has been shown to significantly affect the behavior of particles in the swarm; thus it should not be assumed that these trends hold. This study identifies the benefits of random regrouping and factorization to CPSO based neural network training, and proposes a number of approaches to problem decomposition for use in neural network training. Experiments are performed on 11 problems with sizes ranging from 35 up to 32,811 weights and biases, using a number of general approaches to problem decomposition, and state of the art algorithms taken from the literature. This study found that the impact of factorization and random regrouping on solution quality and swarm behavior depends heavily on the general approach to problem decomposition. It is shown that a random problem decomposition is effective in feed forward neural network training. A random problem decomposition has the benefit of reducing the issue of problem decomposition to the tuning of a single parameter.

Partial Opposition-Based Particle Swarm Optimizer in Artificial Neural Network Training for Medical Data Classification

This paper presents an improved opposition-based Particle Swarm Optimizer (PSO) with partial opposition-based learning. The partial opposition-based learning scheme is a new form of opposition-based learning and it is employed to improve the performance. Nowadays, the artificial neural network (ANN), an important machine learning tool, is used in medicine especially in medical disease diagnosis. ANN training is a complex task and a training algorithm has a significant role in ANN’s performance. Therefore, the proposed algorithm is applied in training of Multi-Layer Feed-Forward Neural Network (FFNN) for classification task in medical data mining. The classification task is carried out on 15 well-known medical data sets. A comparative study has been made with Levenberg–Marquardt algorithm, Artificial Bee Colony, Trigonometric Mutation Differential Evolution, basic PSO and other nine opposition-based PSO algorithms. The experimental results with statistical analysis demonstrate that the proposed algorithm performs better than all other algorithms in classification.

Structure-Preserving Neural Style Transfer.

State-of-the-art neural style transfer methods have demonstrated amazing results by training feed-forward convolutional neural networks or using an iterative optimization strategy. The image representation used in these methods, which contains two components: style representation and content representation, is typically based on high-level features extracted from pretrained classification networks. Because the classification networks are originally designed for object recognition, the extracted features often focus on the central object and neglect other details. As a result, the style textures tend to scatter over the stylized outputs and disrupt the content structures. To address this issue, we present a novel image stylization method that involves an additional structure representation. Our structure representation, which considers two factors: i) the global structure represented by the depth map and ii) the local structure details represented by the image edges, effectively reflects the spatial distribution of all the components in an image as well as the structure of dominant objects respectively. Experimental results demonstrate that our method achieves an impressive visual effectiveness, which is particularly significant when processing images sensitive to structure distortion, e.g. images containing multiple objects potentially at different depths, or dominant objects with clear structures.

Training Feedforward Neural Networks for Structural Health Monitoring of an Aircraft Structure

In the past decades, structural health monitoring (SHM) has become an emerging research area for globally monitoring expensive aircraft and bridge structures. This paper presents the application of stochastic fractal search (SFS) algorithm and its chaotic-enhanced variants to train feedforward neural networks (FNNs) for monitoring an aircraft structure based on vibration data. An experimental spectral testing was carried out to obtain the normal and damaged condition data of a laboratory stiffened panel structure which imitated wingbox of an aircraft. Added mass as pseudo-fault was employed to simulate damage condition with three different damage levels at three different locations. Vibration signature features were generated based on measured frequency response functions (FRFs) and principle component analysis (PCA). Then, metaheuristic-based FNNs approach were applied to localize and predict the severity of damage on the structure. The results reveal that the proposed approach produces high classification and localization accuracy as parameters of the FNNs were optimized systematically using metaheuristic algorithms. In conclusion, the Sine chaos-enhanced SFS algorithm highlights better convergence performance and results accuracy compared to other contested metaheuristic approaches.

Numerical optimization and feed-forward neural networks training using an improved optimization algorithm: multiple leader salp swarm algorithm

Metaheuristics are one of the most promising techniques for solving optimization problems. Salp swarm algorithm (SSA) is a new swarm intelligence based metaheuristic. To improve the performance of SSA, this paper introduces multiple leader salp swarm algorithm (MLSSA), which has more exploratory power than SSA. The algorithm is tested on several mathematical optimization benchmark functions. Results are compared with some well known metaheuristics. The results represents the capability of MLSSA to converge towards the optimum. In recent studies many metaheuristic techniques are applied to train feed-forward neural networks. In this paper MLSSA is also applied for neural network training and is analysed for thirteen different datasets. Performance is compared with SSA, particle swarm optimization, differential evolution, genetic algorithm, ant colony optimization and evolution strategy.

Lifted Proximal Operator Machines

We propose a new optimization method for training feedforward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feedforward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machine (LPOM). LPOM is block multiconvex in all layer-wise weights and activations. This allows us to use block coordinate descent to update the layer-wise weights and activations. Most notably, we only use the mapping of the activation function itself, rather than its derivative, thus avoiding the gradient vanishing or blow-up issues in gradient based training methods. So our method is applicable to various non-decreasing Lipschitz continuous activation functions, which can be saturating and non-differentiable. LPOM does not require more auxiliary variables than the layer-wise activations, thus using roughly the same amount of memory as stochastic gradient descent (SGD) does. Its parameter tuning is also much simpler. We further prove the convergence of updating the layer-wise weights and activations and point out that the optimization could be made parallel by asynchronous update. Experiments on MNIST and CIFAR-10 datasets testify to the advantages of LPOM.

Training Feed-Forward Artificial Neural Networks with a modified artificial bee colony algorithm

Deep learning is a branch of neural network which has been intensively developed in the last decade. Due to the high-accuracy classification ability, the deep learning algorithms have been widely used in many fields, such as speech recognition, image recognition, and natural speech processing. However, they also show some shortcomings especially on the selection of some parameters in the network, including hyper-parameters, which is still treated as a time consuming task. In this paper, a modified ABC (ABC-ISB) optimization algorithm is proposed to automatically train the parameters of Feed-Forward Artificial Neural Networks, which is a typical a neural network. In the proposed ABC algorithm, we utilize the information of neighbors with better performance to accelerate the convergence of employed and onlooker bees respectively. In addition, a new selection strategy and a gbest-guided strategy are introduced to enhance the global search capability and balance the exploration and exploitation of the algorithm separately. The experimental results show our ABC-ISB is generally leading and competitive.

Training of feedforward neural networks for data classification using hybrid particle swarm optimization, Mantegna Lévy flight and neighborhood search

Artificial Neural networks (ANNs) are often applied to data classification problems. However, training ANNs remains a challenging task due to the large and high dimensional nature of search space particularly in the process of fine-tuning the best set of control parameters in terms of weight and bias. Evolutionary algorithms are proved to be a reliable optimization method for training the parameters. While a number of conventional training algorithms have been proposed and applied to various applications, most of them share the common disadvantages of local optima stagnation and slow convergence. In this paper, we propose a new evolutionary training algorithm referred to as LPSONS, which combines the velocity operators in Particle Swarm Optimization (PSO) with Mantegna Lévy distribution to produce more diverse solutions by dividing the population and generation between different sections of the algorithm. It further combines Neighborhood Search with Mantegna Lévy distribution to mitigate premature convergence and avoid local minima. The proposed algorithm can find optimal results and at the same time avoid stagnation in local optimum solutions as well as prevent premature convergence in training Feedforward Multi-Layer Perceptron (MLP) ANNs. Experiments with fourteen standard datasets from UCI machine learning repository confirm that the LPSONS algorithm significantly outperforms a gradient-based approach as well as some well-known evolutionary algorithms that are also based on enhancing PSO.

A FNN for source range and ocean environment classification using time-domain features

Acoustic source ranging in an uncertain ocean environment is a complicated problem, though classification and regression-based machine learning algorithms show promise. A feedforward neural network (FNN) has been trained to do either classification or regression on both the source-receiver range and environment type using extracted time-domain features. Time waveforms are generated to simulate signals received at different ranges in three different environments with a sandy, muddy, or mixed sediment bottom. Four features are extracted from these waveforms: peak level, integrated level, signal length, and later decay time. These four features are used to train FNN for both classification and regression of range and environment type, and the results are compared to a network trained on the time waveforms. For small amounts of training data, the extracted features provide a higher accuracy than the full waveform. Thus, physics-based feature selection via preprocessing can lead to fairly accurate results when using a FNN with small datasets. These results lay a foundation for comparisons to the more computationally expensive convolutional neural networks. [Work supported by the Office of Naval Research.]

Combustion analysis of a compression-ignition engine fuelled with an algae biofuel blend and diethyl ether as an additive by using an artificial neural network

Experimental tests were performed on a single-cylinder, four-stroke, direct-injection, naturally aspirated diesel engine at a constant engine speed of 1500 rpm and at various loading conditions to evaluate its performance, emission, and combustion characteristics. When DEE was added to the algal biofuel blend, the maximum braking thermal efficiency of the engine increased by 7.2% and the minimum brake specific fuel consumption reduced by 6.3%. A reduction in the HC (12%) and CO (19%) emissions was observed with a marginal increase in the NOx (3%) emissions compared with diesel, without any modifications. Moreover, the algae biofuel with diethyl ether (DEE) exhibited combustion results similar to those exhibited by diesel fuel. However, a shortened ignition delay was observed due to their early combustion. The Levenberg–Marquardt backpropagation feedforward neural network (BPNN) training algorithm with a tangent sigmoid transfer function provides an optimum model for effectively predicting engine characteristics. In this study, the overall mean square error, mean absolute percentage error, and regression coefficient for the developed BPNN model were 0.011%, 4.81%, and 0.96, respectively. The experimental results indicated that small content by volume of the algae biofuel blend (BD20) with 5% DEE as a fuel additive exhibited satisfactory results and could be used as a moderate substitute for conventional fuel.

A new evolutionary neural networks based on intrusion detection systems using locust swarm optimization

The need to avoid computer system breaches is increasing. Many researchers have adopted different approaches, such as intrusion detection systems (IDSs), to handle various threats. Intrusion detection has become an imperative system to detect various security breaches. Until today, researchers face the problem of building reliable and effective IDSs that can handle numerous attacks with changing patterns. This paper deals with feed-forward neural network (FNN) training problems using the application of a recently invented meta-heuristic optimization algorithm locust swarm optimization (LSO) for the first time. FNN is combined with LSO (FNN-LSO) to build an advanced detection system and improve the performance of IDS. Our method is applied to a series of experiments to study the capability and performance of the proposed approach. Experimental studies began by using intrusion detection evaluation data, namely, NSL-KDD and UNSW-NB15, to benchmark the performance of the proposed approach. The most common evolutionary trainers, namely, particle swarm optimizer PSO-based trainer and genetic algorithm GA-based trainer, were implemented to verify the results. Compared with existing methods in the literature, our proposed approach provides to be more accurate to be an alternative solution for IDS. The experimental results show that our training algorithm not only attained a very good performance in terms of speed convergence but also achieved reliability due to the reduced likelihood of being trapped in local minima. Furthermore, our proposed model improves the detection rate.

Parameter Self-Adaptation in an Ant Colony Algorithm for Continuous Optimization

ACO $_{\mathbb {R}}$ is a well-established ant colony optimization algorithm for continuous-domain optimization. We present an approach for the dynamic adaptation of the ACO $_{\mathbb {R}}$ algorithm’s controlling parameters, focusing on the search width parameter, based on using several pre-specified parameter configurations, which we call personalities. Before an ant starts to generate a candidate solution, it stochastically adopts a personality based on the relative past success of different personalities. The success of a personality is measured, in turn, by the survival rate of the previous solutions generated by the ants adopting that personality. The premise of our approach is that some personalities will be more appropriate than others for different phases of the search. In addition, our adaptive approach can accommodate solution recombination, the use of which within ACO $_{\mathbb {R}}$ was recently explored in the previous work. It allows the frequency of applying recombination and the type of recombination operator to be dynamically adapted, by having one or more recombination personalities among the competing personalities. We evaluate these proposals experimentally on two applications: 1) training feedforward neural networks for classification using 65 benchmark datasets from the University of California Irvine repository and 2) optimizing several popular synthetic benchmark continuous-domain functions. Our experimental results indicate that our proposals perform better than the standard ACO $_{\mathbb {R}}$ on both applications, to a statistically significant extent.

Optimized Feedforward Neural Network Training for Efficient Brillouin Frequency Shift Retrieval in Fiber

Artificial neural networks (ANNs) can be used to replace traditional methods in various fields, making signal processing more efficient and meeting the real-time processing requirements of the Internet of Things (IoT). As a special type of ANN, recently the feedforward neural network (FNN) has been used to replace the time-consuming Lorentzian curve fitting (LCF) method in Brillouin optical time-domain analysis (BOTDA) to retrieve the Brillouin frequency shift (BFS), which could be used as the indicator in temperature/strain sensing, etc. However, FNN needs to be re-trained if the generalization ability is not satisfactory, or the frequency scanning step is changing in the experiment. This is a cumbersome and inefficient process. In this paper, FNN only needs to be trained once with the proposed method. 150.62 km BOTDA is built to verify the performance of the trained FNN. Simulation and experimental results show that the proposed method is promising in BOTDA because of its high computational efficiency and wide adaptability.

A New Quantum Tunneling Particle Swarm Optimization Algorithm for Training Feedforward Neural Networks

A New Quantum Tunneling Particle Swarm Optimization Algorithm for Training Feedforward Neural Networks

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

Based on a novel algorithm, known as the upper-layer-solution-aware (USA), a new algorithm, in which the penalty method is introduced into the empirical risk, is studied for training feed-forward neural networks in this paper, named as USA with penalty. Both theoretical analysis and numerical results show that it can control the magnitude of weights of the networks. Moreover, the deterministic theoretical analysis of the new algorithm is proved. The monotonicity of the empirical risk with penalty term is guaranteed in the training procedure. The weak and strong convergence results indicate that the gradient of the total error function with respect to weights tends to zero, and the weight sequence goes to a fixed point when the iterations approach positive infinity. Numerical experiment has been implemented and effectively verifies the proved theoretical results.