Training Feedforward Neural Networks Research Articles

An ensemble of differential evolution and Adam for training feed-forward neural networks

Adam is an adaptive gradient descent approach that is commonly used in back-propagation (BP) algorithms for training feed-forward neural networks (FFNNs). However, it has the defect that it may easily fall into local optima. To solve this problem, some metaheuristic approaches have been proposed to train FFNNs. While these approaches have stronger global search capabilities enabling them to more readily escape from local optima, their convergence performance is not as good as that of Adam. The proposed algorithm is an ensemble of differential evolution and Adam (EDEAdam), which integrates a modern version of the differential evolution algorithm with Adam, using two different sub-algorithms to evolve two sub-populations in parallel and thereby achieving good results in both global and local search. Compared with traditional algorithms, the integration of the two algorithms endows EDEAdam with powerful capabilities to handle different classification problems. Experimental results prove that EDEAdam not only exhibits improved global and local search capabilities, but also achieves a fast convergence speed.

Optimized Extreme Learning Machine

Abstract: Extreme Learning Machine (ELM) is a learning method for single-hidden layer feedforward neural network (SLFN) training. The ELM strategy speeds up learning by generating input weights and biases for hidden nodes at random rather than modifying network parameters, making it much faster than the standard gradient-based approach. In this project, an ELM optimized by Hybrid Particle Swarm Optimization approach is presented to optimize the input weights and hidden biases for ELM. We will analyze and obtain results for benchmark datasets. The Optimized Extreme Learning Machine algorithm's output is compared to publicly available data. Later we will compare different algorithms and check which one gives better output metrics. Keywords: ELM, SLFN, PSO, Gradient-based approach, Optimization

New artificial neural network design for Chua chaotic system prediction using FPGA hardware co-simulation

<p>This study aims to design a new architecture of the artificial neural networks (ANNs) using the Xilinx system generator (XSG) and its hardware co-simulation equivalent model using field programmable gate array (FPGA) to predict the behavior of Chua’s chaotic system and use it in hiding information. The work proposed consists of two main sections. In the first section, MATLAB R2016a was used to build a 3×4×3 feed forward neural network (FFNN). The training results demonstrate that FFNN training in the Bayesian regulation algorithm is sufficiently accurate to directly implement. The second section demonstrates the hardware implementation of the network with the XSG on the Xilinx artix7 xc7a100t-1csg324 chip. Finally, the message was first encrypted using a dynamic Chua system and then decrypted using ANN’s chaotic dynamics. ANN models were developed to implement hardware in the FPGA system using the IEEE 754 Single precision floating-point format. The ANN design method illustrated can be extended to other chaotic systems in general.</p>

On the relationship between predictive coding and backpropagation.

Artificial neural networks are often interpreted as abstract models of biological neuronal networks, but they are typically trained using the biologically unrealistic backpropagation algorithm and its variants. Predictive coding has been proposed as a potentially more biologically realistic alternative to backpropagation for training neural networks. This manuscript reviews and extends recent work on the mathematical relationship between predictive coding and backpropagation for training feedforward artificial neural networks on supervised learning tasks. Implications of these results for the interpretation of predictive coding and deep neural networks as models of biological learning are discussed along with a repository of functions, Torch2PC, for performing predictive coding with PyTorch neural network models.

Formation of the quadcopter flight path under overland monitoring using neuro-fuzzy modeling methods

Some territories where economic activity is carried out are characterized by the presence of mountainі and forests. To provide information support for the development of infrastructure and agriculture in these areas, in some cases, there is required overland monitoring with un-manned technologies. In this regard, an algorithm for the formation of a 3D trajectory of a quadcopter during overland piloting in a mountainous and wooded landscape is proposed, which implies autonomous maneuvering to overcome possible obstacles. As a basic model, it is pro-posed to use a Fuzzy Inference System with input characteristics in the form of linguistic varia-bles that reflect fuzzy sectors of space, within which the presence of obstacles and the distance to them are interpreted verbally, i.e., in the form of terms of corresponding input linguistic var-iables. Overcoming obstacles is supposed to be performed on the basis of fuzzy conclusions of the proposed system, formulated as terms of output linguistic variables which reflect changes in the angle of rotation in the horizontal plane, flight altitude, and traverse speed of the quadcop-ter. The paper analyzes the results of the model behavior for different scenarios of the terms of the input linguistic variables. For the operational formation of the quadcopter flight path, it is also proposed to use neural network modeling tools. Due to its ability to adapt to new condi-tions and requirements, the neural network model can become an important tool in ensuring the autonomous flight of a quadcopter under overland monitoring. For appropriate training of the three-layer feedforward neural network, a sufficiently large number of quadcopter behavior scenarios are used, which were generated by the Fuzzy Inference System relative to the over-coming of possible obstacles in five sectors of the survey.

An Adaptive Sigmoidal Activation Function for Training Feed Forward Neural Network Equalizer

Feed for word neural networks (FFNN) have attracted a great attention, in digital communication area. Especially they are investigated as nonlinear equalizers at the receiver, to mitigate channel distortions and additive noise. The major drawback of the FFNN is their extensive training. We present a new approach to enhance their training efficiency by adapting the activation function. Adapting procedure for activation function extensively increases the flexibility and the nonlinear approximation capability of FFNN. Consequently, the learning process presents better performances, offers more flexibility and enhances nonlinear capability of NN structure thus the final state kept away from undesired saturation regions. The effectiveness of the proposed method is demonstrated through different challenging channel models, it performs quite well for nonlinear channels which are severe and hard to equalize. The performance is measured throughout, convergence properties, minimum bit error achieved. The proposed algorithm was found to converge rapidly, and accomplish the minimum steady state value. All simulation shows that the proposed method improves significantly the training efficiency of FFNN based equalizer compared to the standard training one.

Using the modified k-mean algorithm with an improved teaching-learning-based optimization algorithm for feedforward neural network training

In this paper we proposed a novel procedure for training a feedforward neural network. The accuracy of artificial neural network outputs after determining the proper structure for each problem depends on choosing the appropriate method for determining the best weights, which is the appropriate training algorithm. If the training algorithm starts from a good starting point, it is several steps closer to achieving global optimization. In this paper, we present an optimization strategy for selecting the initial population and determining the optimal weights with the aim of minimizing neural network error. Teaching-learning-based optimization (TLBO) is a less parametric algorithm rather than other evolutionary algorithms, so it is easier to implement. We have improved this algorithm to increase efficiency and balance between global and local search. The improved teaching-learning-based optimization (ITLBO) algorithm has added the concept of neighborhood to the basic algorithm, which improves the ability of global search. Using an initial population that includes the best cluster centers after clustering with the modified k-mean algorithm also helps the algorithm to achieve global optimum. The results are promising, close to optimal, and better than other approach which we compared our proposed algorithm with them.

Performance Assessment of a Novel Eco-Friendly Solar Panel Mounted Hybrid Rotating Energy System with Renewable Energy Applications

It is a known fact that fossil fuels will be depleted in the near future owing to the negative effects on the environment. The number of applications using renewable energy sources instead of fossil fuels to obtain energy has increased significantly. It is aimed to provide effective energy production with the proposed rotary energy system (RES) installation for regions with high wind energy and solar energy potential. In the paper, the design, manufacturing process, installation, and output power prediction of a novel RES are presented. In the proposed system, a hybrid system whose energy is derived from solar and wind energy is envisaged. The electrical characteristics of the solar panels with dimensions 140 × 60 × 2.5 mm mounted on the prototype are 6 V 100 mA. The prototype has been tested at different rotation speeds to evaluate the effect of wind energy. Moreover, the output power prediction based on Feedforward Neural Network (FFNN) and Particle Swarm Optimization trained Feedforward Neural Network (PSO-FFNN) has been performed with the data obtained from the prototype system. The three quantitative standard statistical performance evaluation measures, root mean square error (RMSE), mean absolute percentage error (MAPE) and Theil's inequality coefficient (TIC) are employed to compare the performances of these architectures. FFNN architecture, the RMSE, MAPE and TIC values are calculated as 0.0690, 0.0455 and 0.0278, respectively. For the PSO-FFNN architecture, RMSE, MAPE and TIC values are 0.0530, 0.0383, and 0.0213, respectively. It has been proved that it will be produced energy more effectively thanks to the hybrid RES in meeting energy demand.

Artificial neural network based meta-heuristic for performance improvement in physical internet supply chain network

<p>Nowadays, reducing total costs while enhancing customer satisfaction is a major task for many supply chain systems. To deal with this issue, the physical internet (PI) paradigm can be represented as a potential replacement for the current logistics system. This paper devoted the cost reduction and lead time improvement in a PI-SCN using a hybrid framework based on an artificial neural network (ANN) and an improved slime mould algorithm (ISMA). To address the performance of the proposed framework, a real-case study in Morocco is considered. The new trainer ISMA’s performance has been investigated in three approximation datasets from the University of California at Irvine (UCI) machine-learning repository regarding nine recent metaheuristics. The experimental results highlight the effectiveness of ISMA according to other meta heuristics for training feed-forward neural networks (FNNs) to converge speed and to avoid local minima.</p>

Comparison of data selection methods for modeling chemical processes with artificial neural networks

Instance selection aims at selecting model training data in a way that the performance of the trained models is maximized. In the context of modeling chemical processes by artificial neural networks, it can serve as an essential preprocessing step since measurement data of such processes are commonly highly clustered and thus far away from being ideally normally distributed. In this paper, four filter methods from literature and a newly proposed method for data selection are tested and combined with a convex hull data selection algorithm, which results in ten different selection approaches. These approaches are applied to five selected datasets by training feed-forward artificial neural networks with the produced split datasets. The final mean model deviation is used to quantify the algorithms’ performance and their standard deviation to provide information about their reproducibility. It is found that the convex hull extended algorithms self-organizing maps based stratified sampling with a proportional allocation rule and the newly proposed self-information-based subset selection perform best for real-world chemical engineering data.

A Novel Fast Feedforward Neural Networks Training Algorithm

Abstract In this paper1 a new neural networks training algorithm is presented. The algorithm originates from the Recursive Least Squares (RLS) method commonly used in adaptive filtering. It uses the QR decomposition in conjunction with the Givens rotations for solving a normal equation - resulting from minimization of the loss function. An important parameter in neural networks is training time. Many commonly used algorithms require a big number of iterations in order to achieve a satisfactory outcome while other algorithms are effective only for small neural networks. The proposed solution is characterized by a very short convergence time compared to the well-known backpropagation method and its variants. The paper contains a complete mathematical derivation of the proposed algorithm. There are presented extensive simulation results using various benchmarks including function approximation, classification, encoder, and parity problems. Obtained results show the advantages of the featured algorithm which outperforms commonly used recent state-of-the-art neural networks training algorithms, including the Adam optimizer and the Nesterov’s accelerated gradient.

Artificial neural network for the recognition of human emotions under a backpropagation algorithm

The era of the technological revolution increasingly encourages the development of technologies that facilitate in one way or another people's daily activities, thus generating a great advance in information processing. The purpose of this work is to implement a neural network that allows classifying the emotional states of a person based on the different human gestures. A database is used with information on students from the PUCE-E School of Computer Science and Engineering. Said information are images that express the gestures of the students and with which the comparative analysis with the input data is carried out. The environment in which this work converges proposes that the implementation of this project be carried out under the programming of a multilayer neuralnetwork. Multilayer feeding neural networks possess a number of properties that make them particularly suitable for complex pattern classification problems [8]. Back-Propagation [4], which is a backpropagation algorithm used in the Feedforward neural network, was taken into consideration to solve the classification of emotions.
 Keywords: Image processing, neural networks, gestures, back-propagation, feedforward, classification, emotions.
 References
 [1]S. Gangwar, S. Shukla, D. Arora. “Human Emotion Recognition by Using Pattern Recognition Network”, Journal of Engineering Research and Applications, Vol. 3, Issue 5, pp.535-539, 2013.
 [2]K. Rohit. “Back Propagation Neural Network based Emotion Recognition System”. International Journal of Engineering Trends and Technology (IJETT), Vol. 22, Nº 4, 2015.
 [3]S. Eishu, K. Ranju, S. Malika, “Speech Emotion Recognition using BFO and BPNN”, International Journal of Advances in Science and Technology (IJAST), ISSN2348-5426, Vol. 2 Issue 3, 2014.
 [4]A. Fiszelew, R. García-Martínez and T. de Buenos Aires. “Generación automática de redes neuronales con ajuste de parámetros basado en algoritmos genéticos”. Revista del Instituto Tecnológico de Buenos Aires, 26, 76-101, 2002.
 [5]Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. “Handwritten digit recognition with a back-propagation network”. In Advances in neural information processing systems. pp. 396-404, 1990.
 [6]G. Bebis and M. Georgiopoulos. “Feed-forward neural networks”. IEEE Potentials, 13(4), 27-31, 1994.
 [7]G. Huang, Q. Zhu and C. Siew. “Extreme learning machine: a new learning scheme of feedforward neural networks”. In Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference. Vol. 2, pp. 985-990. IEEE, 2004.
 [8]D. Montana and L. Davis. “Training Feedforward Neural Networks Using Genetic Algorithms”. In IJCAI, Vol. 89, pp. 762-767, 1989.
 [9]I. Sutskever, O. Vinyals and Q. Le. “Sequence to sequence learning with neural networks”. In Advances in neural information processing systems. pp. 3104-3112, 2014.
 [10]J. Schmidhuber. “Deep learning in neural networks: An overview”. Neural networks, 61, 85-117, 2015.
 [11]R. Santos, M. Ruppb, S. Bonzi and A. Filetia, “Comparación entre redes neuronales feedforward de múltiples capas y una red de función radial para detectar y localizar fugas en tuberías que transportan gas”. Chem. Ing.Trans , 32 (1375), e1380, 2013.

Modeling and Predicting the Cell Migration Properties from Scratch Wound Healing Assay on Cisplatin-Resistant Ovarian Cancer Cell Lines Using Artificial Neural Network.

The study of artificial neural networks (ANN) has undergone a tremendous revolution in recent years, boosted by deep learning tools. The presence of a greater number of learning tools and their applications, in particular, favors this revolution. However, there is a significant need to deal with the issue of implementing a systematic method during the development phase of the ANN to increase its performance. A multilayer feedforward neural network (FNN) was proposed in this paper to predict the cell migration assay on cisplatin-sensitive and cisplatin-resistant (CisR) ovarian cancer (OC) cell lines via scratch wound healing assay. An FNN training algorithm model was generated using the MATLAB fitting function in a MATLAB script to accomplish this task. The input parameters were types of cell lines, times, and wound area, and outputs were relative wound area, percentage of wound closure, and wound healing speed. In addition, we tested and compared the initial accuracy of various supervised learning classifier and support vector regression (SVR) algorithms. The proposed ANN model achieved good agreement with the experimental data and minimized error between the estimated and experimental values. The conclusions drawn demonstrate that the developed ANN model is a useful, accurate, fast, and inexpensive method to predict cancerous cell migration characteristics evaluated via scratch wound healing assay.

A Self-Adaptive Hybrid Bat Algorithm for Training Feedforward Neural Networks

Training feedforward neural network (FFNN) is a complex task in the supervised learning field. An FFNN trainer aims to find the best set of weights that minimizes classification error. This paper presents a new training method based on hybrid bat optimization with self-adaptive differential evolution to train the feedforward neural networks. The hybrid training algorithm combines bat and the self-adaptive differential evolution algorithm called BAT-SDE. BAT-SDE is used to better search in the solution space, which proves its effectiveness in large space solutions. The performance of the proposed approach was compared with eight evolutionary techniques and the standard momentum backpropagation and adaptive learning rate. The comparison was benchmarked and evaluated using seven bio-medical datasets and one large credit card fraud detection dataset. The results of the comparative study show that BAT-SDE outperformed other training methods in most datasets and can be an alternative to other training methods.

Novel sensitivity method for evaluating the first derivative of the feed-forward neural network outputs

Evaluating the exact first derivative of a feedforward neural network (FFNN) output with respect to the input feature is pivotal for performing the sensitivity analysis of the trained neural network with respect to the inputs. In this paper, a novel method is presented that computes the analytical quality first derivative of a trained feedforward neural network output with respect to the input features without the need for backpropagation. To this end, the complex step derivative approximation is illustrated, and its implementation in the framework of the feedforward neural network is described. Artificial datasets are generated, and the efficacy of the proposed method for both regression and classification tasks is demonstrated. The results obtained for the regression task indicated that the proposed method is capable of obtaining analytical quality derivatives, and in the case of the classification task, the least relevant features could be identified.

A novel hybrid deep learning method with cuckoo search algorithm for classification of arrhythmia disease using ECG signals

This work presents an efficient hybridized approach for the classification of electrocardiogram (ECG) samples into crucial arrhythmia classes to detect heartbeat abnormalities. The physiological detection using electrocardiogram (ECG) signals has been the most popular means and widely accepted automated detection system to monitor heart health. Additionally, arrhythmia beat classification plays a prominent role in electrocardiogram (ECG) analysis dedicated elucidate cardiac health status while analyzing heart rhythm. The authors aim to classify ECG samples into major arrhythmia classes precisely by removing the inherent noise of ECG signals in preprocessing phase using discrete wavelet transformation (DWT). The QRS complex plays a crucial role in ECG signal identification. Therefore, the position and amplitude of R-peaks are determined to detect the QRS complex. The feature vectors of the QRS complex are further optimized with cuckoo search (CS) optimization algorithm in addition to denoising signals using DWT to select the most relevant set of features. The Support vector machine (SVM)-trained support vector contains the best training information used to train feed-forward back-propagation neural network (FFBPNN) to propose the variant DWT + CS + SVM-FFBPNN to classify signals among five classes. MIT-BIH arrhythmia database is utilized for different types of heartbeats. The classification analysis based on a variant with optimized feature vector using cuckoo search algorithm and SVM-FFBPNN determines heart rate with an accuracy of 98.319%. In contrast, the variant FFBPNN without optimization obtains 97.95% accuracy. The improved performance of the novel combination of classifiers resulted in overall classification accuracy of 98.53% with precision and recall of 98.247% and 95.68%, respectively. The simulation analysis comprising 3600 samples and 1160 heartbeats also outperformed the existing arrhythmia classifications performed based on neural networks. This illustrates the success of the proposed ECG classification model in accurately categorizing ECG signals for arrhythmia classification.

Neural-Network-Based Multiobjective Optimizer for Dual-Band Circularly Polarized Antenna

A multiobjective optimization (MOO) technique for a dual-band circularly polarized antenna by using neural networks (NNs) is introduced in this paper. In particular, the optimum antenna dimensions are computed by modeling the problem as a multilayer feed-forward neural network (FFNN), which is two-stage trained with I/O pairs. The FFNN is chosen because of its characteristic of accurate approximation and good generalization. The data for FFNN training is obtained by using HFSS EM simulator by varying different geometrical parameters of the antenna. A two strip-loaded circular aperture antenna is utilized to demonstrate the optimization technique. The target dual bands are 835–865 MHz and 2.3–2.35 GHz.

Estimation of tumor parameters using neural networks for inverse bioheat problem

Background and objectiveSome types of cancer cause rapid cell growth, while others cause cells to grow and divide at a slower rate. Certain forms of cancer result in visible growths called tumors. This work proposes an inverse estimation of the size and location of the tumor using a feedforward Neural Network (FFNN) model. MethodsThe forward model is a 3D model of the breast induced with a tumor of various sizes at different locations within the breast, and it is solved using the Pennes equation. The data obtained from the simulation of the bioheat transfer is used for training the neural network. In order to optimize the neural network architecture, the work proposes varying the number of neurons in the hidden layer and thus finding the best fit to create a relationship between the temperature profile and tumor parameters which can be used to estimate the tumor parameters given the temperature profile. ResultsThese simulations resulted in a temperature distribution profile that could thus be used to locate and determine the parameters of the cancerous tumor within the breast. The prediction accuracy showed the capacity of the trained Feed Forward Neural Network to estimate the unknown parameters within an acceptable range of error. The model validations use the Root Mean Square Error method to quantify and minimize the prediction error. ConclusionsIn this work, a non-intrusive method for the diagnosis of breast cancer was modelled, which yields conclusive results for the estimation of the tumor parameters.

Data-driven Prediction of Temperature Evolution in Metallic Additive Manufacturing Process

In this study, a data-driven deep learning model for fast and accurate prediction of temperature evolution and melting pool size of metallic additive manufacturing processes are developed. The study focuses on bulk experiments of the M4 high-speed steel material powder manufactured by Direct Energy Deposition. Under non-optimized process parameters, many deposited layers (above 30) generate large changes of microstructure through the sample depth caused by the high sensitivity of the cladding material on the thermal history. A 2D finite element analysis (FEA) of the bulk sample, validated in a previous study by experimental measurements, is able to achieve numerical data defining the temperature field evolution under different process settings. A Feed-forward neural networks (FFNN) approach is trained to reproduce the temperature fields generated from FEA. Hence, the trained FFNN is used to predict the history of the temperature fields for new process parameter sets not included in the initial dataset. Besides the input energy, nodal coordinates, and time, five additional features relating layer number, laser location, and distance from the laser to sampling point are considered to enhance prediction accuracy. The results indicate that the temperature evolution is predicted well by the FFNN with an accuracy of 99% within 12 seconds.

Vortex search optimization algorithm for training of feed-forward neural network

Training of feed-forward neural-networks (FNN) is a challenging nonlinear task in supervised learning systems. Further, derivative learning-based methods are frequently inadequate for the training phase and cause a high computational complexity due to the numerous weight values that need to be tuned. In this study, training of neural-networks is considered as an optimization process and the best values of weights and biases in the structure of FNN are determined by Vortex Search (VS) algorithm. The VS algorithm is a novel metaheuristic optimization method recently developed, inspired by the vortex shape of stirred liquids. VS fulfills the training task to set the optimal weights and biases stated in a matrix. In this context, the proposed VS-based learning method for FNNs (VS-FNN) is conducted to analyze the effectiveness of the VS algorithm in FNN training for the first time in the literature. The proposed method is applied to six datasets whose names are 3-bit XOR, Iris Classification, Wine-Recognition, Wisconsin-Breast-Cancer, Pima-Indians-Diabetes, and Thyroid-Disease. The performance of the proposed algorithm is analyzed by comparing with other training methods based on Artificial Bee Colony Optimization (ABC), Particle Swarm Optimization (PSO), Simulated Annealing (SA), Genetic Algorithm (GA) and Stochastic Gradient Descent (SGD) algorithms. The experimental results show that VS-FNN is generally leading and competitive. It is also said that VS-FNN can be used as a capable tool for neural networks.