Feed-forward Neural Network Algorithm Research Articles

Optimal Randomness for Stochastic Configuration Network (SCN) with Heavy-Tailed Distributions.

Stochastic Configuration Network (SCN) has a powerful capability for regression and classification analysis. Traditionally, it is quite challenging to correctly determine an appropriate architecture for a neural network so that the trained model can achieve excellent performance for both learning and generalization. Compared with the known randomized learning algorithms for single hidden layer feed-forward neural networks, such as Randomized Radial Basis Function (RBF) Networks and Random Vector Functional-link (RVFL), the SCN randomly assigns the input weights and biases of the hidden nodes in a supervisory mechanism. Since the parameters in the hidden layers are randomly generated in uniform distribution, hypothetically, there is optimal randomness. Heavy-tailed distribution has shown optimal randomness in an unknown environment for finding some targets. Therefore, in this research, the authors used heavy-tailed distributions to randomly initialize weights and biases to see if the new SCN models can achieve better performance than the original SCN. Heavy-tailed distributions, such as Lévy distribution, Cauchy distribution, and Weibull distribution, have been used. Since some mixed distributions show heavy-tailed properties, the mixed Gaussian and Laplace distributions were also studied in this research work. Experimental results showed improved performance for SCN with heavy-tailed distributions. For the regression model, SCN-Lévy, SCN-Mixture, SCN-Cauchy, and SCN-Weibull used less hidden nodes to achieve similar performance with SCN. For the classification model, SCN-Mixture, SCN-Lévy, and SCN-Cauchy have higher test accuracy of 91.5%, 91.7% and 92.4%, respectively. Both are higher than the test accuracy of the original SCN.

Machine Learning for Ionic Liquid Toxicity Prediction

In addition to proper physicochemical properties, low toxicity is also desirable when seeking suitable ionic liquids (ILs) for specific applications. In this context, machine learning (ML) models were developed to predict the IL toxicity in leukemia rat cell line (IPC-81) based on an extended experimental dataset. Following a systematic procedure including framework construction, hyper-parameter optimization, model training, and evaluation, the feedforward neural network (FNN) and support vector machine (SVM) algorithms were adopted to predict the toxicity of ILs directly from their molecular structures. Based on the ML structures optimized by the five-fold cross validation, two ML models were established and evaluated using IL structural descriptors as inputs. It was observed that both models exhibited high predictive accuracy, with the SVM model observed to be slightly better than the FNN model. For the SVM model, the determination coefficients were 0.9289 and 0.9202 for the training and test sets, respectively. The satisfactory predictive performance and generalization ability make our models useful for the computer-aided molecular design (CAMD) of environmentally friendly ILs.

Classification of thought evoked potentials for navigation and communication using multilayer neural network

ABSTRACT In this paper, an intelligent classification system has been developed to command a robot chair by means of direct brain activity, aided by amplification. The intelligent system classifies seven fundamental tasks based on measuring ElectroEncephaloGraphic (EEG) brain activity. The seven tasks were used to control a robot chair and also to interact with others. In this analysis, a simple protocol for the EEG data acquisition procedure has been proposed to perform seven tasks based on thought evoked potentials (TEP’s). The evoked potentials were converted into control signals to navigate the robot chair and also to choose words/letters in an oddball paradigm for communication. In the EEG acquiring process, five volunteers participated and brain activities related to navigational movements (Forward, Left, and Right) and communication (Yes, No, and Help) were recorded from the volunteers to form the database. The acquired EEG signals are visually validated upon recording each trial and pre-processed to eliminate the noise contents. The pre-processed signals were segmented into six frequency bands to extract spectral band energy and spectral band centroid features. The extracted features were then formed to classify the tasks using a feed-forward Multilayer Neural Network algorithm to exhibit customized (subject wise) features. The trained models of the neural networks were compared to validate the classification results. From the results, it is observed that the Spectral centroid features have the highest classification rate of 98.50%.

A Bilevel Learning Model and Algorithm for Self-Organizing Feed-Forward Neural Networks for Pattern Classification.

Conventional artificial neural network (ANN) learning algorithms for classification tasks, either derivative-based optimization algorithms or derivative-free optimization algorithms work by training ANN first (or training and validating ANN) and then testing ANN, which are a two-stage and one-pass learning mechanism. Thus, this learning mechanism may not guarantee the generalization ability of a trained ANN. In this article, a novel bilevel learning model is constructed for self-organizing feed-forward neural network (FFNN), in which the training and testing processes are integrated into a unified framework. In this bilevel model, the upper level optimization problem is built for testing error on testing data set and network architecture based on network complexity, whereas the lower level optimization problem is constructed for network weights based on training error on training data set. For the bilevel framework, an interactive learning algorithm is proposed to optimize the architecture and weights of an FFNN with consideration of both training error and testing error. In this interactive learning algorithm, a hybrid binary particle swarm optimization (BPSO) taken as an upper level optimizer is used to self-organize network architecture, whereas the Levenberg-Marquardt (LM) algorithm as a lower level optimizer is utilized to optimize the connection weights of an FFNN. The bilevel learning model and algorithm have been tested on 20 benchmark classification problems. Experimental results demonstrate that the bilevel learning algorithm can significantly produce more compact FFNNs with more excellent generalization ability when compared with conventional learning algorithms.

Quantum Algorithms for Feedforward Neural Networks

Quantum machine learning has the potential for broad industrial applications, and the development of quantum algorithms for improving the performance of neural networks is of particular interest given the central role they play in machine learning today. We present quantum algorithms for training and evaluating feedforward neural networks based on the canonical classical feedforward and backpropagation algorithms. Our algorithms rely on an efficient quantum subroutine for approximating inner products between vectors in a robust way, and on implicitly storing intermediate values in quantum random access memory for fast retrieval at later stages. The running times of our algorithms can be quadratically faster in the size of the network than their standard classical counterparts since they depend linearly on the number of neurons in the network, and not on the number of connections between neurons. Furthermore, networks trained by our quantum algorithm may have an intrinsic resilience to overfitting, as the algorithm naturally mimics the effects of classical techniques used to regularize networks. Our algorithms can also be used as the basis for new quantum-inspired classical algorithms with the same dependence on the network dimensions as their quantum counterparts but with quadratic overhead in other parameters that makes them relatively impractical.

Using spotted hyena optimizer for training feedforward neural networks

Spotted hyena optimizer (SHO) is a novel metaheuristic optimization algorithm based on the behavior of spotted hyena and their collaborative behavior in nature. In this paper, we design a spotted hyena optimizer for training feedforward neural network (FNN), which is regarded as a challenging task since it is easy to fall into local optima. Our objective is to apply metaheuristic optimization algorithm to tackle this problem better than the mathematical and deterministic methods. In order to confirm that using SHO to train FNN is more effective, five classification datasets and three function-approximations are applied to benchmark the performance of the proposed method. The experimental results show that the proposed SHO algorithm for optimization FNN has the best comprehensive performance and has more outstanding performance than other the state-of-the-art metaheuristic algorithms in terms of the performance measures.

A Hybrid Method Based on Extreme Learning Machine and Self Organizing Map for Pattern Classification.

Extreme learning machine is a fast learning algorithm for single hidden layer feedforward neural network. However, an improper number of hidden neurons and random parameters have a great effect on the performance of the extreme learning machine. In order to select a suitable number of hidden neurons, this paper proposes a novel hybrid learning based on a two-step process. First, the parameters of hidden layer are adjusted by a self-organized learning algorithm. Next, the weights matrix of the output layer is determined using the Moore–Penrose inverse method. Nine classification datasets are considered to demonstrate the efficiency of the proposed approach compared with original extreme learning machine, Tikhonov regularization optimally pruned extreme learning machine, and backpropagation algorithms. The results show that the proposed method is fast and produces better accuracy and generalization performances.

Neighborhood centroid opposite-based learning Harris Hawks optimization for training neural networks

The Harris Hawks Optimization Algorithm is a new metaheuristic optimization that simulates the process of Harris Hawk hunting prey (rabbit) in nature. The global and local search processes of the algorithm are performed by simulating several stages of cooperative behavior during hunting. To enhance the performance of this algorithm, in this paper we propose a neighborhood centroid opposite-based learning Harris Hawks optimization algorithm (NCOHHO). The mechanism of applying the neighborhood centroid under the premise of using opposite-based learning technology to improve the performance of the algorithm, the neighborhood centroid is used as a reference point for the generation of the opposite particle, while maintaining the diversity of the population and make full use of the swarm search experience to expand the search range of the reverse solution. Enhancing the probability of finding the optimal solution and the improved algorithm is superior to the original Harris Hawks Optimization algorithm in all aspects. We apply NCOHHO to the training of feed-forward neural network (FNN). To confirm that using NCOHHO to train FNN is more effective, five classification datasets are applied to benchmark the performance of the proposed method. Comprehensive comparison and analysis from the three aspects of mean, variance and classification success rate, the experimental results show that the proposed NCOHHO algorithm for optimization FNN has the best comprehensive performance and has more outstanding performance than other metaheuristic algorithms in terms of the performance measures.

Memetic algorithms for training feedforward neural networks: an approach based on gravitational search algorithm

The backpropagation (BP) algorithm is a gradient-based algorithm used for training a feedforward neural network (FNN). Despite the fact that BP is still used today when FNNs are trained, it has some disadvantages, including the following: (i) it fails when non-differentiable functions are addressed, (ii) it can become trapped in local minima, and (iii) it has slow convergence. In order to solve some of these problems, metaheuristic algorithms have been used to train FNN. Although they have good exploration skills, they are not as good as gradient-based algorithms at exploitation tasks. The main contribution of this article lies in its application of novel memetic approaches based on the Gravitational Search Algorithm (GSA) and Chaotic Gravitational Search Algorithm (CGSA) algorithms, called respectively Memetic Gravitational Search Algorithm (MGSA) and Memetic Chaotic Gravitational Search Algorithm (MCGSA), to train FNNs in three classical benchmark problems: the XOR problem, the approximation of a continuous function, and classification tasks. The results show that both approaches constitute suitable alternatives for training FNNs, even improving on the performance of other state-of-the-art metaheuristic algorithms such as ParticleSwarm Optimization (PSO), the Genetic Algorithm (GA), the Adaptive Differential Evolution algorithm with Repaired crossover rate (Rcr-JADE), and the Covariance matrix learning and Bimodal distribution parameter setting Differential Evolution (COBIDE) algorithm. Swarm optimization, the genetic algorithm, the adaptive differential evolution algorithm with repaired crossover rate, and the covariance matrix learning and bimodal distribution parameter setting differential evolution algorithm.

Congestion Prediction System With Artificial Neural Networks

Software Defined Network (SDN) is a programmable network architecture that provides innovative solutions to the problems of the traditional networks. Congestion control is still an uncharted territory for this technology. In this work, a congestion prediction scheme has been developed by using neural networks. Minimum Redundancy Maximum Relevance (mRMR) feature selection algorithm was performed on the data collected from the OMNET++ simulation. The novelty of this study also covers the implementation of mRMR in an SDN congestion prediction problem. After evaluating the relevance scores, two highest ranking features were used. On the learning stage Nonlinear Autoregressive Exogenous Neural Network (NARX), Nonlinear Autoregressive Neural Network, and Nonlinear Feedforward Neural Network algorithms were executed. These algorithms had not been used before in SDNs according to the best of the authors knowledge. The experiments represented that NARX was the best prediction algorithm. This machine learning approach can be easily integrated to different topologies and application areas.

Quantitative assessment of forest ecosystem stress caused by cement plant pollution using in situ measurements and Sentinel-2 satellite data in a part of the UNESCO World Heritage Site

Anthropogenic industrial dust decreases productivity and slows down the growth of plants. Quantifying the effects of industrial dust on vegetation and determining the distance over which factories scatter dust are of paramount importance for biodiversity conservation and sustaining ecosystem services. This study aims at quantifying the effect of dust emitted by the Neka cement plant (NCP), Mazandaran province, northern Iran, on the surrounding Hyrcanian forests based on an analysis of the Leaf Area Index (LAI) retrieved from Sentinel-2 imagery. An Inductively Coupled Plasma Mass Spectrometer (ICP-MS) was used to quantify the concentrations of cadmium (Cd), chromium (Cr), copper (Cu), lead (Pb), calcium (Ca), magnesium (Mg), sodium (Na), silicon (Si) and zinc (Zn) in leaves of the dominant Chestnut-leaved Oak (Quercus castaneifolia). A feed-forward neural network algorithm and field measurements were used to retrieve the leaf area index (LAI) from Sentinel-2 data with a RMSE of 0.42 (m2/m2). MODIS-NDVI and EVI time series spanning 17 years (2000 to 2017) were analysed to ensure the independence of the variation in the condition of the vegetation from drought or other environmental factors. The results indicate that Sentinel-2 data can be used to map degradation due to pollution from the cement plant and quantify the effect of the dust spatially. Dust from the cement plant (dust source) was carried approximately 4700 meters in the direction of the prevailing wind. A significant correlation of 0.849 was recorded between LAI and distance from the NCP. It is concluded that dust from the NCP had adverse ecological effects on the neighbouring forest ecosystems recently designated a UNESCO World Heritage Site.

An efficient background calibration technique for analog-to-digital converters based on neural network

This paper introduces a background digital calibration algorithm based on neural network, which can adaptively calibrate multiple non-ideal factors in a single-channel ADC, such as gain error, mismatch, offset and harmonic distortion. It enables an efficient background calibration through a simple feed forward neural network and LM gradient descent algorithm. The simulation results show that in the case of a signal input close to the Nyquist frequency, for a 14-bit 500 MS/s prototype ADC, only about 40,000 data needed, the ENOB of the ADC can be increased from 7.81 to 13.06 and the SFDR from 49.7 dB to 106.8 dB assisted by a lower speed reference ADC.

Design and Implementation of Intelligent Hybrid Neuro Controller for Automatic Voltage Regulator

In this paper, we presented a hybrid controller by combining the advantages of both PID and Neuro controllers for automatic voltage regulator (AVR) system. A Neuro controller is designed using multilayer feedforward neural network and Levenberg-Marquardt backpropagation algorithm is used for training the network. Also, hybrid controller is achieved by blending the characteristics of classical PID and proposed Neuro controller using a switching mechanism based on the error. The proposed PID, Neuro and Hybrid controllers are simulated in MATLAB environment and their transient response parameters are compared. The simulation results clearly indicated the improvement in the transient output of the automatic voltage regulator system with proposed hybrid controller even in the presence of uncertainties in the system.

An online self-organizing algorithm for feedforward neural network

Feedforward neural network (FNN) is the most popular network model, and the appropriate structure and learning algorithms are the key of its performance. This paper proposes an online self-organizing algorithm for feedforward neural network (OSNN) with a single hidden layer. The proposed OSNN optimizes the structure of FNN for time-varying system including structure design and parameter learning. In structure design, this paper measures the contribution ratios of hidden nodes by local sensitivity analysis based on differentiation method. OSNN merges hidden nodes with the others that have the highest correlation when their contribution ratios are almost zero and adds new hidden nodes by error reparation. For parameter learning, an improved online gradient method (OGM), called online gradient method with fixed memory (FMOGM), is proposed to improve the convergence speed and accuracy of OGM. In addition, this paper calculates the contribution ratios and the network error and estimates the local minima by using the fixed-sized training set of FMOGM instead of one sample at the current time, which can obtain more effective local information and a compact network structure. Finally, the proposed OSNN is verified using a number of benchmark problems and a practical problem for biochemical oxygen demand prediction in wastewater treatment. The experimental results show that OSNN has better convergence speed and accuracy than other algorithms.

Insights into Potent Therapeutical Antileukemic Agent L-glutaminase Enzyme Under Solid-state Fermentation: A Review.

To review the applications and production studies of reported antileukemic drug L-glutaminase under Solid-state Fermentation (SSF). An amidohydrolase that gained economic importance because of its wide range of applications in the pharmaceutical industry, as well as the food industry, is L-glutaminase. The medical applications utilized it as an anti-tumor agent as well as an antiretroviral agent. L-glutaminase is employed in the food industry as an acrylamide degradation agent, as a flavor enhancer and for the synthesis of theanine. Another application includes its use in hybridoma technology as a biosensing agent. Because of its diverse applications, scientists are now focusing on enhancing the production and optimization of L-glutaminase from various sources by both Solid-state Fermentation (SSF) and submerged fermentation studies. Of both types of fermentation processes, SSF has gained importance because of its minimal cost and energy requirement. L-glutaminase can be produced by SSF from both bacteria and fungi. Single-factor studies, as well as multi-level optimization studies, were employed to enhance L-glutaminase production. It was concluded that L-glutaminase activity achieved by SSF was 1690 U/g using wheat bran and Bengal gram husk by applying feed-forward artificial neural network and genetic algorithm. The highest L-glutaminase activity achieved under SSF was 3300 U/gds from Bacillus sp., by mixture design. Purification and kinetics studies were also reported to find the molecular weight as well as the stability of L-glutaminase. The current review is focused on the production of L-glutaminase by SSF from both bacteria and fungi. It was concluded from reported literature that optimization studies enhanced L-glutaminase production. Researchers have also confirmed antileukemic and anti-tumor properties of the purified L-glutaminase on various cell lines.

Implementation of artificial neural networks for very short range weather prediction

Weather conditions are a significant factor for various sectors such as transportation safety, development, health, etc. Therefore, high development is needed in forecasting future weather conditions. Many ways are used to predict weather conditions. Along with the development of technology now, weather prediction can be made using Artificial Intelligence (AI) technology or artificial intelligence so that the results obtained are more optimal. In this study, the artificial neural network used has a feedforward neural network algorithm using training data consisting of temperature, air pressure, air humidity, wind speed, hourly wind speed at the Juanda Meteorological Station in Surabaya in Januari 2019 with the target is intensity of rainfall. Furthermore, the data was released in the period of 1 January 2019 to 31 Januari 2019. Based on the analysis results, the Artificial Neural Network model has a fairly good performance in predicting an increase in rainfall in Surabaya. The best model is considered by a model with architecture 7 - 60 - 1 with an estimate correlation is 0.87, with an error value of -0.03. With this model, it is expected to become one of the forecaster considerations in making special weather forecasts at intervals every hour.

Equilibrium, kinetics and artificial intelligence characteristic analysis for Zn (II) ion adsorption on rice husks digested with nitric acid

The use of low-cost adsorbents produced from rice husks that can selectively remove zinc (II) ion from wastewater effluent has been investigated. The disorder and fatal diseases can be triggered in living organisms by accumulating the heavy metals in the wastewater. Adsorption is an operative cleansing and separation method used in the industry to remove contaminants from effluents. In order to convert the rice husk to the adsorbent, digestion and carbonization processes were performed. Rice husks digested nitric acid at 1.0 M to reduce the organic constituents and further carbonized at 400, 600 and 800 °C to eliminate the non-carbon element in rice husks. The physiochemical characteristics of the digested and carbonized rice husks were analyzed using scanning electron microscope (SEM), X-ray diffraction (XRD) and atomic adsorption spectrophotometer (AAS). The SEM morphological characteristics showed the high pyrolysis effect on the organic matter left the samples with more pores and cavities, which enhanced the surface area and uptake, especially for 800 °C carbonized samples. The samples’ physical characteristics using XRD showed that the treatment with HNO3 illustrated a pattern close to the activated carbon with slightly lower intensities. The applied feed-forward back-propagation neural network algorithm showed high validity (R2 ≈ 0.9686) for the tested experimental data sets and hence can be applied for forthcoming studies with nearby conditions. The outcomes showed that the increment of the carbonization temperature is correlated with the contact time and can increase the metal uptake percentage and adsorption capacity. The adsorption followed the Langmuir isotherm, which indicates monolayer coverage. The data showed that adsorption kinetics follow the pseudo-second-order rate expression.

Data-Driven Modeling of Smartphone-Based Electrochemiluminescence Sensor Data Using Artificial Intelligence.

Understanding relationships among multimodal data extracted from a smartphone-based electrochemiluminescence (ECL) sensor is crucial for the development of low-cost point-of-care diagnostic devices. In this work, artificial intelligence (AI) algorithms such as random forest (RF) and feedforward neural network (FNN) are used to quantitatively investigate the relationships between the concentration of luminophore and its experimentally measured ECL and electrochemical data. A smartphone-based ECL sensor with /TPrA was developed using disposable screen-printed carbon electrodes. ECL images and amperograms were simultaneously obtained following 1.2-V voltage application. These multimodal data were analyzed by RF and FNN algorithms, which allowed the prediction of concentration using multiple key features. High correlation (0.99 and 0.96 for RF and FNN, respectively) between actual and predicted values was achieved in the detection range between 0.02 µM and 2.5 µM. The AI approaches using RF and FNN were capable of directly inferring the concentration of using easily observable key features. The results demonstrate that data-driven AI algorithms are effective in analyzing the multimodal ECL sensor data. Therefore, these AI algorithms can be an essential part of the modeling arsenal with successful application in ECL sensor data modeling.

Integration of synthetic aperture radar and optical satellite data for corn biomass estimation

Efforts to use satellites to monitor the condition and productivity of crops, although extensive, can be challenging to operationalize at field scales in part due to low frequency revisit of higher resolution space-based sensors, in the context of an actively growing crop canopy. The presence of clouds and cloud shadows further impedes the exploitation of high resolution optical sensors for operational monitoring of crop development. The objective of this research was to present an option to facilitate greater temporal observing opportunities to monitor the accumulation of corn biomass, by integrating biomass products from Synthetic Aperture Radar (SAR) and optical satellite sensors. To accomplish this integration, a transfer function was developed using a Neural Network algorithm to relate estimated corn biomass from SAR to that estimated from optical data. With this approach, end users can exploit biomass products to monitor corn development, regardless of the source of satellite data.•The Water Cloud Model (WCM) was calibrated or parametrized for horizontal transmit and horizontal received (HH) and horizontal transmit and vertical received (HV) C-band SAR backscatter using a least square algorithm.•Biomass and volumetric soil moisture were estimated from dual-polarized RADARSAT-2 images without any ancillary input data.•A feed forward backpropagation Neural Network algorithm was trained as a transfer function between the biomass estimates from RADARSAT-2 and the biomass estimates from RapidEye.

Machine learning facilitated business intelligence (Part I)

PurposeThe purpose of this paper is to conduct a comprehensive review of the noteworthy contributions made in the area of the Feedforward neural network (FNN) to improve its generalization performance and convergence rate (learning speed); to identify new research directions that will help researchers to design new, simple and efficient algorithms and users to implement optimal designed FNNs for solving complex problems; and to explore the wide applications of the reviewed FNN algorithms in solving real-world management, engineering and health sciences problems and demonstrate the advantages of these algorithms in enhancing decision making for practical operations.Design/methodology/approachThe FNN has gained much popularity during the last three decades. Therefore, the authors have focused on algorithms proposed during the last three decades. The selected databases were searched with popular keywords: “generalization performance,” “learning rate,” “overfitting” and “fixed and cascade architecture.” Combinations of the keywords were also used to get more relevant results. Duplicated articles in the databases, non-English language, and matched keywords but out of scope, were discarded.FindingsThe authors studied a total of 80 articles and classified them into six categories according to the nature of the algorithms proposed in these articles which aimed at improving the generalization performance and convergence rate of FNNs. To review and discuss all the six categories would result in the paper being too long. Therefore, the authors further divided the six categories into two parts (i.e. Part I and Part II). The current paper, Part I, investigates two categories that focus on learning algorithms (i.e. gradient learning algorithms for network training and gradient-free learning algorithms). Furthermore, the remaining four categories which mainly explore optimization techniques are reviewed in Part II (i.e. optimization algorithms for learning rate, bias and variance (underfitting and overfitting) minimization algorithms, constructive topology neural networks and metaheuristic search algorithms). For the sake of simplicity, the paper entitled “Machine learning facilitated business intelligence (Part II): Neural networks optimization techniques and applications” is referred to as Part II. This results in a division of 80 articles into 38 and 42 for Part I and Part II, respectively. After discussing the FNN algorithms with their technical merits and limitations, along with real-world management, engineering and health sciences applications for each individual category, the authors suggest seven (three in Part I and other four in Part II) new future directions which can contribute to strengthening the literature.Research limitations/implicationsThe FNN contributions are numerous and cannot be covered in a single study. The authors remain focused on learning algorithms and optimization techniques, along with their application to real-world problems, proposing to improve the generalization performance and convergence rate of FNNs with the characteristics of computing optimal hyperparameters, connection weights, hidden units, selecting an appropriate network architecture rather than trial and error approaches and avoiding overfitting.Practical implicationsThis study will help researchers and practitioners to deeply understand the existing algorithms merits of FNNs with limitations, research gaps, application areas and changes in research studies in the last three decades. Moreover, the user, after having in-depth knowledge by understanding the applications of algorithms in the real world, may apply appropriate FNN algorithms to get optimal results in the shortest possible time, with less effort, for their specific application area problems.Originality/valueThe existing literature surveys are limited in scope due to comparative study of the algorithms, studying algorithms application areas and focusing on specific techniques. This implies that the existing surveys are focused on studying some specific algorithms or their applications (e.g. pruning algorithms, constructive algorithms, etc.). In this work, the authors propose a comprehensive review of different categories, along with their real-world applications, that may affect FNN generalization performance and convergence rate. This makes the classification scheme novel and significant.