Training Feedforward Neural Networks Research Articles

Mapping of Eucalyptus in Natura 2000 Areas Using Sentinel 2 Imagery and Artificial Neural Networks

Plantations of fast-growing Eucalyptus trees have become a common sight in the western Iberian peninsula where they are planted to exploit their economic potential. Negative side-effects of large scale plantations including the invasive behavior of Eucalyptus trees outside of regular plantations have become apparent. This study uses medium resolution, multi-spectral imagery of the Sentinel 2 satellites to map Eucalyptus across Portugal and parts of Spain with a focus on Natura 2000 areas inside Portugal, that are protected under the European birds and habitats directives. This method enables the detection of small incipient as well as mixed populations outside of regular plantations. Ground truth maps were compiled using field surveys as well as high resolution satellite imagery and were used to train Feedforward Neural Networks. These models predict Eucalyptus tree cover with a sensitivity of up to 75.7% as well as a specificity of up to 95.8%. The overall accuracy of the prediction is 92.5%. A qualitative assessment of Natura 2000 areas in Portugal has been performed and 15 areas have been found to be affected by Eucalyptus of which 9 are strongly affected. This study demonstrates the applicability of multi-spectral imagery for tree-species classification and invasive species control. It provides a probability-map of Eucalyptus tree cover for the western Iberian peninsula with 10 m spatial resolution and shows the need for monitoring of Eucalyptus in protected areas.

Development of New Rheological Models for Class G Cement with Nanoclay as an Additive Using Machine Learning Techniques.

The rheology of the oil well cement plays a pivotal role in the cement placement. Accurate prediction of cement rheological parameters helps to monitor the durability and pumpability of the cement slurry. In this study, an artificial neural network is used to develop different models for the prediction of various rheological parameters such as shear stress, apparent viscosity, plastic viscosity, and yield point of a class G cement slurry with nanoclay as an additive. An extensive experimental study was conducted to generate enough data set for the training of artificial intelligence models. The class G oil well cement slurries were prepared by fixing the water–cement ratio to 0.44 and adding organically modified nanoclays as a strength enhancer. The rheological properties of the oil well cement slurries were investigated at a wide range of temperatures (37 ≤ T ≤ 90 °C) and shear rates (5 ≤ γ ≤ 500 s–1). Experimental data generated were used for the training of feed-forward neural networks. The predicted values of the rheological properties from the trained model showed a good agreement when compared with the experimental values. The average absolute percentage error was less than 5% in both training and validation phases of modeling. A trend analysis was carried out to ensure that the proposed models can define the underlying physics. From the validation and the trend analysis, it was found that the new models can be used to predict cement rheological properties within the range of data set on which the models were trained. The proposed models are independent of laboratory-dependent variables and can give quick and real-time values of the rheological parameters.

Memetic algorithms for training feedforward neural networks: an approach based on gravitational search algorithm

The backpropagation (BP) algorithm is a gradient-based algorithm used for training a feedforward neural network (FNN). Despite the fact that BP is still used today when FNNs are trained, it has some disadvantages, including the following: (i) it fails when non-differentiable functions are addressed, (ii) it can become trapped in local minima, and (iii) it has slow convergence. In order to solve some of these problems, metaheuristic algorithms have been used to train FNN. Although they have good exploration skills, they are not as good as gradient-based algorithms at exploitation tasks. The main contribution of this article lies in its application of novel memetic approaches based on the Gravitational Search Algorithm (GSA) and Chaotic Gravitational Search Algorithm (CGSA) algorithms, called respectively Memetic Gravitational Search Algorithm (MGSA) and Memetic Chaotic Gravitational Search Algorithm (MCGSA), to train FNNs in three classical benchmark problems: the XOR problem, the approximation of a continuous function, and classification tasks. The results show that both approaches constitute suitable alternatives for training FNNs, even improving on the performance of other state-of-the-art metaheuristic algorithms such as ParticleSwarm Optimization (PSO), the Genetic Algorithm (GA), the Adaptive Differential Evolution algorithm with Repaired crossover rate (Rcr-JADE), and the Covariance matrix learning and Bimodal distribution parameter setting Differential Evolution (COBIDE) algorithm. Swarm optimization, the genetic algorithm, the adaptive differential evolution algorithm with repaired crossover rate, and the covariance matrix learning and bimodal distribution parameter setting differential evolution algorithm.

Incorporating Dynamic Production-Logging Data to the Permeability-Estimation Workflow Using Machine Learning

Summary The objective of this work is to develop and train feedforward artificial neural networks (ANNs) on the forecasting of layer permeability in heterogeneous reservoirs. The results are validated by comparing the model outputs with permeability curves computed from production logging data. Production logs are used as targets to train the model. A flow-profile interpretation method is used to compute continuous permeability curves free of wellbore skin effects. In addition, segmentation techniques are applied to high-resolution ultrasonic image logs. These logs provide not only the image of the mega- and giga-pore system but can also identify the permeable facies along the reservoir. The image segmentation jointly with other borehole logs provides the necessary features for the network training process. The proposed neural network focuses on delivering reliable and validated permeability curves. Its development accounts for formation skin factor, as well as nongeological noise usually found in ultrasonic image logs. The procedure is tested on both synthetic and field data sets. The estimations presented herein demonstrate the model's ability to learn nonlinear relationships between geological input variables and reservoir dynamic data even if the actual physical relationship is complex and not known a priori. Although the preprocessing stages of the procedure involve some expertise in data interpretation, the neural-network structure can be easily coded in any programming language, requiring no assumptions on physics in advance. For the case studies presented in this work, the proposed procedure provides more accurate permeability curves than the ones obtained from conventional methods, which usually fail to predict the permeability measured on drill-stem tests conducted in dual-porosity reservoirs. The novelty of this work is to incorporate dynamic production-logging (PL) data into the permeability-estimation workflow. Correction Notice: The preprint paper was updated from its originally published version to correct Fig. 17 on page 11. An erratum detailing the change is included in the Supporting Information section below.

CLS Next Gen: Accurate Frequency-Frequency Correlation Functions from Center Line Slope Analysis of 2D Correlation Spectra Using Artificial Neural Networks.

The center line slope (CLS) observable has become a popular method for characterizing spectral diffusion dynamics in two-dimensional (2D) correlation spectroscopy because of its ease of implementation, robustness, and clear theoretical relationship to the frequency-frequency correlation function (FFCF). The FFCF relates the frequency fluctuations of an ensemble of chromophores to coupled bath modes of the chemical system and is used for comparison to molecular dynamics simulations and for calculating 2D spectra. While in the appropriate limits, the CLS can be shown to be the normalized FFCF, from which the full FFCF can be obtained, in practice the assumptions that relate the CLS to the normalized FFCF are frequently violated. These violations are due to the presence of homogeneous broadening and motional narrowing. The generalized problem of relating the CLS to the FFCF is reanalyzed by introducing a new set of dimensionless parameters for both the CLS and FFCF. A large data set was generated of CLS parameters derived from numerically modeled 2D line shapes with known FFCF parameters. This data set was used to train feedforward artificial neural networks that act as functions, which take the CLS parameters as inputs and return FFCF parameters. These neural networks were deployed in an algorithm that is able to quickly and accurately determine FFCF parameters from experimental CLS parameters and the fwhm of the absorption line shape. The method and necessary inputs to accurately obtain the FFCF from the CLS are presented.

Training feedforward neural network using genetic algorithm to diagnose left ventricular hypertrophy

In this research work, a new technique was proposed for the diagnosis of left ventricular hypertrophy (LVH) from the ECG signal. The advanced imaging techniques can be used to diagnose left ventricular hypertrophy, but it leads to time-consuming and more expensive. This proposed technique overcomes thesef issues and may serve as an efficient tool to diagnose the LVH disease. The LVH causes changes in the patterns of ECG signal which includes R wave, QRS and T wave. This proposed approach identifies the changes in the pattern and extracts the temporal, spatial and statistical features of the ECG signal using windowed filtering technique. These features were applied to the conventional classifier and also to the neural network classifier with the modified weights using a genetic algorithm. The weights were modified by combining the crossover operators such as crossover arithmetic and crossover two-point operator. The results were compared with the various classifiers and the performance of the neural network with the modified weights using a genetic algorithm is outperformed. The accuracy of the weights modified feedforward neural network is 97.5%.

Evolutionary optimization of artificial neural network using an interactive phase-based optimization algorithm for chaotic time series prediction

The prediction of chaotic time series is an important issue in nonlinear information procession. Due to the multi-modal, high-dimensional and non-differentiable or discontinuous characteristics of chaotic systems, global optimization techniques are required to avoid from falling into local optima for the prediction of chaotic time series. Phase-based optimization is recently proposed as a global search algorithm inspired by natural phenomena. In this paper, an improved phase-based optimization algorithm integrating stochastic interaction strategy and global optimal interaction strategy, termed interactive phase-based optimization (IPBO), is proposed to train feed-forward neural networks (FNNs) for chaotic time series prediction. The combination of stochastic interaction strategy and global optimal interaction strategy can balance the capability of exploration and exploitation in the global optimization process. To demonstrate the searching capability, sixteen widely used benchmark functions are firstly used to investigate its optimization performance. Then, the prediction effectiveness of FNNs trained by IPBO has been illustrated using classical chaotic time series of Lorenz, Box–Jenkins and Mackey–Glass. The training and testing performances of IPBO and other state-of-the-art optimization algorithms have been compared for predicting these time series. Conducted numerical experiments indicate that IPBO is not only competitive in functions optimization and has also a better learning ability in training FNNs among other state-of-the-art optimization algorithms.

Enhanced Salp Swarm Algorithm based on random walk and its application to training feedforward neural networks

Salp Swarm Algorithm (SSA) is a new type of metaheuristic and has shown superiority over other well-known algorithms such as Particle Swarm Optimization and Grey Wolf Optimizer in solving challenging optimization problems. Despite its superior performance, SSA still has problems such as insufficient convergence speed. Moreover, its local optima avoidance ability is not as good as those evolutionary algorithms using crossover operators. In this paper, we propose a modified Salp Swarm Algorithm (m-SSA) which improves the exploitation and exploration of SSA by integrating random walk strategy and especially enhances exploration by adding a new controlling parameter. In addition, a simulated annealing-type acceptance criterion is adopted to accept the fittest follower position as the new best leader position. The performance of the proposed algorithm is benchmarked on a set of classical functions and CEC2014 test suite. The proposed algorithm (m-SSA) outperforms SSA significantly on most test functions. When compared with other state-of-the-art metaheuristics, it also presents very competitive results. Besides, we apply the proposed algorithm on training feedforward neural networks (FNNs) and the results prove the effectiveness and efficiency of m-SSA.

A New Hybrid Grasshopper Optimization - Backpropagation for Feedforward Neural Network Training

The Grasshopper optimization algorithm showed a rapid converge in the initial phases of the global search, however while being around the global optimum, the searching process became so slow. On the contrary, the gradient descending method around achieved faster convergent speed global optimum, and the convergent accuracy was showed to be higher at the same time. As a result, the proposed hybrid algorithm combined Grasshopper optimization algorithm (GOA) along with the back-propagation (BP) algorithm, also referred to as GOA–BP algorithm, was introduced to provide training to the weights of the feed forward neural network (FNN), the proposed hybrid algorithm can utilize the strong global searching ability of the GOA, and the intense local searching ability of the Back-Propagation algorithm. The results of experiments showed that the proposed hybrid GOA–BP algorithm was better and faster in convergent speed and accuracy than the Grasshopper optimization algorithm (GOA) and BP algorithm.
 
 http://dx.doi.org/10.25130/tjps.25.2020.018

Multi-Start Method with Cutting for Solving Problems of Unconditional Optimization

In the present paper, we present a multi-start method with dynamic cutting of “unpromising” starts of locally optimal algorithms for solving continuous unconditional optimization problems. We also describe our results of testing of the proposed method for solving problems of feed-forward neural network training.

Wall-modeled large-eddy simulations of spanwise rotating turbulent channels—Comparing a physics-based approach and a data-based approach

We develop wall modeling capabilities for large-eddy simulations (LESs) of channel flow subjected to spanwise rotation. The developed models are used for flows at various Reynolds numbers and rotation numbers, with different grid resolutions and in differently sized computational domains. We compare a physics-based approach and a data-based machine learning approach. When pursuing a data-based approach, we use the available direct numerical simulation data as our training data. We highlight the difference between LES wall modeling, where one writes all flow quantities in a coordinate defined by the wall-normal direction and the near-wall flow direction, and Reynolds-averaged Navier-Stokes modeling, where one writes flow quantities in tensor forms. Pursuing a physics-based approach, we account for system rotation by reformulating the eddy viscosity in the wall model. Employing the reformulated eddy viscosity, the wall model is able to predict the mean flow correctly. Pursuing a data-based approach, we train a fully connected feed-forward neural network (FNN). The FNN is informed about our knowledge (although limited) on the mean flow. We then use the trained FNNs as wall models in wall modeled LES (WMLES) and show that it predicts the mean flow correctly. While it is not the focus of this study, special attention is paid to the problem of log-layer mismatch, which is common in WMLES. Our study shows that log-layer mismatch, or rather, linear-layer mismatch in WMLES of spanwise rotating channels, is not present at high rotation numbers, even when the wall-model/LES matching location is at the first grid point.

Performance comparison of feedforward neural network training algorithms in modelling microwave pyrolysis of oil palm fibre for hydrogen and biochar production

AbstractMicrowave pyrolysis of oil palm fibre (OPF) was conducted to study the effect of microwave power, temperature, and nitrogen (N2) flow rate on the production of hydrogen and biochar. The effect of microwave power ranging from 400 to 900 W, reaction temperatures ranging from 450°C to 700°C, and N2 flow rates ranging 200 to 1,200 cm3 min−1 were investigated. The microwave power, reaction temperature, and N2 flow rates are directly proportional to hydrogen and inversely proportional to biochar yield. The characteristics of the biochar were analysed using CHNOS, scanning electron microscopy, and Brunauer–Emmett–Teller results. Central composite design was used to optimize the number of experiment. The optimized experimental datasets were used as inputs for developing a model that can predict the weight of hydrogen and biochar from microwave pyrolysis of OPF. A multilayer feedforward neural network model was developed by comparing the performance of 11 different types of backpropagation training algorithms. The 11 training algorithms belong to six classes, which are (a) additive momentum, (b) self‐adaptive learning rate, (c) resilient backpropagation, (d) conjugate gradient, (e) quasi‐Newton, and (f) Bayesian regulation (BR). Best performing training algorithm was selected based on the lowest error values computed. The objective of this research is to identify the most suitable training algorithm for this process. Levenberg–Marquardt and BR exhibited very good performance. BR exhibited best performance with lowest error values in predicting weight of hydrogen and biochar. The testing data error values for hydrogen weight is 0.216 root mean square error (RMSE), 0.144 mean absolute error (MAE), and 0.020 mean absolute percentage error (MAPE), and biochar is 0.886 RMSE, 0.489 MAE, and 0.019 MAPE. The optimum number of hidden neuron for this algorithm is 19 with mean square error 0.23. The biochar exhibits porous structure suggesting possible land application.

Machine Learning, Urban Water Resources Management and Operating Policy

Meticulously analyzing all contemporaneous conditions and available options before taking operations decisions regarding the management of the urban water resources is a necessary step owing to water scarcity. More often than not, this analysis is challenging because of the uncertainty regarding inflows to the system. The most common approach to account for this uncertainty is to combine the Bayesian decision theory with the dynamic programming optimization method. However, dynamic programming is plagued by the curse of dimensionality, that is, the complexity of the method is proportional to the number of discretized possible system states raised to the power of the number of reservoirs. Furthermore, classical statistics does not consistently represent the stochastic structure of the inflows (see persistence). To avoid these problems, this study will employ an appropriate stochastic model to produce synthetic time-series with long-term persistence, optimize the system employing a network flow programming modelling, and use the optimization results for training a feedforward neural network (FFN). This trained FFN alone can serve as a decision support tool that describes not only reservoir releases but also how to operate the entire water supply system. This methodology is applied in a simplified representation of the Athens water supply system, and the results suggest that the FFN is capable of successfully operating the system according to a predefined operating policy.

Random Regrouping and Factorization in Cooperative Particle Swarm Optimization Based Large-Scale Neural Network Training

Previous studies have shown that factorization and random regrouping significantly improve the performance of the cooperative particle swarm optimization (CPSO) algorithm. However, few studies have examined whether this trend continues when CPSO is applied to the training of feed forward neural networks. Neural network training problems often have very high dimensionality and introduce the issue of saturation, which has been shown to significantly affect the behavior of particles in the swarm; thus it should not be assumed that these trends hold. This study identifies the benefits of random regrouping and factorization to CPSO based neural network training, and proposes a number of approaches to problem decomposition for use in neural network training. Experiments are performed on 11 problems with sizes ranging from 35 up to 32,811 weights and biases, using a number of general approaches to problem decomposition, and state of the art algorithms taken from the literature. This study found that the impact of factorization and random regrouping on solution quality and swarm behavior depends heavily on the general approach to problem decomposition. It is shown that a random problem decomposition is effective in feed forward neural network training. A random problem decomposition has the benefit of reducing the issue of problem decomposition to the tuning of a single parameter.

Partial Opposition-Based Particle Swarm Optimizer in Artificial Neural Network Training for Medical Data Classification

This paper presents an improved opposition-based Particle Swarm Optimizer (PSO) with partial opposition-based learning. The partial opposition-based learning scheme is a new form of opposition-based learning and it is employed to improve the performance. Nowadays, the artificial neural network (ANN), an important machine learning tool, is used in medicine especially in medical disease diagnosis. ANN training is a complex task and a training algorithm has a significant role in ANN’s performance. Therefore, the proposed algorithm is applied in training of Multi-Layer Feed-Forward Neural Network (FFNN) for classification task in medical data mining. The classification task is carried out on 15 well-known medical data sets. A comparative study has been made with Levenberg–Marquardt algorithm, Artificial Bee Colony, Trigonometric Mutation Differential Evolution, basic PSO and other nine opposition-based PSO algorithms. The experimental results with statistical analysis demonstrate that the proposed algorithm performs better than all other algorithms in classification.

Structure-Preserving Neural Style Transfer.

State-of-the-art neural style transfer methods have demonstrated amazing results by training feed-forward convolutional neural networks or using an iterative optimization strategy. The image representation used in these methods, which contains two components: style representation and content representation, is typically based on high-level features extracted from pretrained classification networks. Because the classification networks are originally designed for object recognition, the extracted features often focus on the central object and neglect other details. As a result, the style textures tend to scatter over the stylized outputs and disrupt the content structures. To address this issue, we present a novel image stylization method that involves an additional structure representation. Our structure representation, which considers two factors: i) the global structure represented by the depth map and ii) the local structure details represented by the image edges, effectively reflects the spatial distribution of all the components in an image as well as the structure of dominant objects respectively. Experimental results demonstrate that our method achieves an impressive visual effectiveness, which is particularly significant when processing images sensitive to structure distortion, e.g. images containing multiple objects potentially at different depths, or dominant objects with clear structures.

Training Feedforward Neural Networks for Structural Health Monitoring of an Aircraft Structure

In the past decades, structural health monitoring (SHM) has become an emerging research area for globally monitoring expensive aircraft and bridge structures. This paper presents the application of stochastic fractal search (SFS) algorithm and its chaotic-enhanced variants to train feedforward neural networks (FNNs) for monitoring an aircraft structure based on vibration data. An experimental spectral testing was carried out to obtain the normal and damaged condition data of a laboratory stiffened panel structure which imitated wingbox of an aircraft. Added mass as pseudo-fault was employed to simulate damage condition with three different damage levels at three different locations. Vibration signature features were generated based on measured frequency response functions (FRFs) and principle component analysis (PCA). Then, metaheuristic-based FNNs approach were applied to localize and predict the severity of damage on the structure. The results reveal that the proposed approach produces high classification and localization accuracy as parameters of the FNNs were optimized systematically using metaheuristic algorithms. In conclusion, the Sine chaos-enhanced SFS algorithm highlights better convergence performance and results accuracy compared to other contested metaheuristic approaches.

Numerical optimization and feed-forward neural networks training using an improved optimization algorithm: multiple leader salp swarm algorithm

Metaheuristics are one of the most promising techniques for solving optimization problems. Salp swarm algorithm (SSA) is a new swarm intelligence based metaheuristic. To improve the performance of SSA, this paper introduces multiple leader salp swarm algorithm (MLSSA), which has more exploratory power than SSA. The algorithm is tested on several mathematical optimization benchmark functions. Results are compared with some well known metaheuristics. The results represents the capability of MLSSA to converge towards the optimum. In recent studies many metaheuristic techniques are applied to train feed-forward neural networks. In this paper MLSSA is also applied for neural network training and is analysed for thirteen different datasets. Performance is compared with SSA, particle swarm optimization, differential evolution, genetic algorithm, ant colony optimization and evolution strategy.

Lifted Proximal Operator Machines

We propose a new optimization method for training feedforward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feedforward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machine (LPOM). LPOM is block multiconvex in all layer-wise weights and activations. This allows us to use block coordinate descent to update the layer-wise weights and activations. Most notably, we only use the mapping of the activation function itself, rather than its derivative, thus avoiding the gradient vanishing or blow-up issues in gradient based training methods. So our method is applicable to various non-decreasing Lipschitz continuous activation functions, which can be saturating and non-differentiable. LPOM does not require more auxiliary variables than the layer-wise activations, thus using roughly the same amount of memory as stochastic gradient descent (SGD) does. Its parameter tuning is also much simpler. We further prove the convergence of updating the layer-wise weights and activations and point out that the optimization could be made parallel by asynchronous update. Experiments on MNIST and CIFAR-10 datasets testify to the advantages of LPOM.

Training Feed-Forward Artificial Neural Networks with a modified artificial bee colony algorithm

Deep learning is a branch of neural network which has been intensively developed in the last decade. Due to the high-accuracy classification ability, the deep learning algorithms have been widely used in many fields, such as speech recognition, image recognition, and natural speech processing. However, they also show some shortcomings especially on the selection of some parameters in the network, including hyper-parameters, which is still treated as a time consuming task. In this paper, a modified ABC (ABC-ISB) optimization algorithm is proposed to automatically train the parameters of Feed-Forward Artificial Neural Networks, which is a typical a neural network. In the proposed ABC algorithm, we utilize the information of neighbors with better performance to accelerate the convergence of employed and onlooker bees respectively. In addition, a new selection strategy and a gbest-guided strategy are introduced to enhance the global search capability and balance the exploration and exploitation of the algorithm separately. The experimental results show our ABC-ISB is generally leading and competitive.