Multilayer Perceptron Training Research Articles

Training Multi-Layer Perceptron with Enhanced Brain Storm Optimization Metaheuristics

In the domain of artificial neural networks, the learning process represents one of the most challenging tasks. Since the classification accuracy highly depends on the weights and biases, it is crucial to find its optimal or suboptimal values for the problem at hand. However, to a very large search space, it is very difficult to find the proper values of connection weights and biases. Employing traditional optimization algorithms for this issue leads to slow convergence and it is prone to get stuck in the local optima. Most commonly, back-propagation is used for multi-layer-perceptron training and it can lead to vanishing gradient issue. As an alternative approach, stochastic optimization algorithms, such as nature-inspired metaheuristics are more reliable for complex optimization tax, such as finding the proper values of weights and biases for neural network training. In this work, we propose an enhanced brain storm optimization-based algorithm for training neural networks. In the simulations, ten binary classification benchmark datasets with different difficulty levels are used to evaluate the efficiency of the proposed enhanced brain storm optimization algorithm. The results show that the proposed approach is very promising in this domain and it achieved better results than other state-of-the-art approaches on the majority of datasets in terms of classification accuracy and convergence speed, due to the capability of balancing the intensification and diversification and avoiding the local minima. The proposed approach obtained the best accuracy on eight out of ten observed dataset, outperforming all other algorithms by 1–2% on average. When mean accuracy is observed, the proposed algorithm dominated on nine out of ten datasets.

An Improved Gorilla Troops Optimizer Based on Lens Opposition-Based Learning and Adaptive β-Hill Climbing for Global Optimization

Gorilla troops optimizer (GTO) is a newly developed meta-heuristic algorithm, which is inspired by the collective lifestyle and social intelligence of gorillas. Similar to other metaheuristics, the convergence accuracy and stability of GTO will deteriorate when the optimization problems to be solved become more complex and flexible. To overcome these defects and achieve better performance, this paper proposes an improved gorilla troops optimizer (IGTO). First, Circle chaotic mapping is introduced to initialize the positions of gorillas, which facilitates the population diversity and establishes a good foundation for global search. Then, in order to avoid getting trapped in the local optimum, the lens opposition-based learning mechanism is adopted to expand the search ranges. Besides, a novel local search-based algorithm, namely adaptive β-hill climbing, is amalgamated with GTO to increase the final solution precision. Attributed to three improvements, the exploration and exploitation capabilities of the basic GTO are greatly enhanced. The performance of the proposed algorithm is comprehensively evaluated and analyzed on 19 classical benchmark functions. The numerical and statistical results demonstrate that IGTO can provide better solution quality, local optimum avoidance, and robustness compared with the basic GTO and five other wellknown algorithms. Moreover, the applicability of IGTO is further proved through resolving four engineering design problems and training multilayer perceptron. The experimental results suggest that IGTO exhibits remarkable competitive performance and promising prospects in real-world tasks.

Feature Selection by Multiobjective Optimization: Application to Spam Detection System by Neural Networks and Grasshopper Optimization Algorithm

Networks are strained by spam, which also overloads email servers and blocks mailboxes with unwanted messages and files. Setting the protective level for spam filtering might become even more crucial for email users when malicious steps are taken since they must deal with an increase in the number of valid communications being marked as spam. By finding patterns in email communications, spam detection systems (SDS) have been developed to keep track of spammers and filter email activity. SDS has also enhanced the tool for detecting spam by reducing the rate of false positives and increasing the accuracy of detection. The difficulty with spam classifiers is the abundance of features. The importance of feature selection (FS) comes from its role in directing the feature selection algorithm’s search for ways to improve the SDS’s classification performance and accuracy. As a means of enhancing the performance of the SDS, we use a wrapper technique in this study that is based on the multi-objective grasshopper optimization algorithm (MOGOA) for feature extraction and the recently revised EGOA algorithm for multilayer perceptron (MLP) training. The suggested system’s performance was verified using the SpamBase, SpamAssassin, and UK-2011 datasets. Our research showed that our novel approach outperformed a variety of established practices in the literature by as much as 97.5%, 98.3%, and 96.4% respectively.

Cyber Intrusion Detection System Based on a Multiobjective Binary Bat Algorithm for Feature Selection and Enhanced Bat Algorithm for Parameter Optimization in Neural Networks

The staggering development of cyber threats has propelled experts, professionals and specialists in the field of security into the development of more dependable protection systems, including effective intrusion detection system (IDS) mechanisms which are equipped for boosting accurately detected threats and limiting erroneously detected threats simultaneously. Nonetheless, the proficiency of the IDS framework depends essentially on extracted features from network traffic and an effective classifier of the traffic into abnormal or normal traffic. The prime impetus of this study is to increase the performance of the IDS on networks by building a two-phase framework to reinforce and subsequently enhance detection rate and diminish the rate of false alarm. The initial stage utilizes the developed algorithm of a proficient wrapper-approach-based feature selection which is created on a multi-objective BAT algorithm (MOBBAT). The subsequent stage utilizes the features obtained from the initial stage to categorize the traffic based on the newly upgraded BAT algorithm (EBAT) for training multilayer perceptron (EBATMLP), to improve the IDS performance. The resulting methodology is known as the (MOB-EBATMLP). The efficiency of our proposition has been assessed by utilizing the mainstream benchmarked datasets: NLS-KDD, ISCX2012, UNSW-NB15, KDD CUP 1999, and CICIDS2017 which are established as standard datasets for evaluating IDS. The outcome of our experimental analysis demonstrates a noteworthy advancement in network IDS above other techniques.

Forecasting the Compressibility Parameters of Gypseous Soil using Artificial Neural Networks

To ensure safe design of structures against settlement, it is necessary to determine the compressibility parameters of the underneath soil especially compression and rebound indices. In this paper, an approach to forecast the compressibility parameters of gypseous soils based on index parameters was developed using Artificial Neural Networks technique. Two equations were developed to estimate compression and rebound indices using back propagation algorithm to train multi-layer perceptron, in which good agreements were achieved. The input parameters used were: the depth, gypsum content, liquid limit, plastic limit, plasticity index, passing sieve No.200, dry unit weight, water content and initial void ratio. Two output parameters were determined including compression index and rebound index. A parametric study was also conducted to investigate the generalization and robustness of both models. The findings indicate that both models were reliable within the range of utilized data. It was found that gypsum content has the highest effect on the compressibility index followed by water content, plasticity index, dry unit weight and plastic limit, while other parameters have lower effect. The gypsum content has the highest effect again on the rebound index followed by passing sieve No.200, initial void ratio, plastic limit and plasticity index, while other parameters have lower effect.

Multi-Layer Perceptron Training Optimization Using Nature Inspired Computing

Although the multi-layer perceptron (MLP) neural networks provide a lot of flexibility and have proven useful and reliable in a wide range of classification and regression problems, they still have limitations. One of the most common is associated with the optimization algorithm used to train them. The most commonly used training method is stochastic gradient descent with backpropagation (or backpropagation for short) because it is mathematically tractable (given that the activation functions are differentiable). However, backpropagation is not guaranteed to find the globally optimal set of weights and biases. As a result, the MLP is often incapable of obtaining a desirable solution to the problem. Clonal selection algorithms (CSA) are optimization procedures that effectively explore a complex and large space to find values near the global optimum. Consequently, CSA can be used to solve the problem of training MLP networks. This paper presents a novel implementation of CSA for training MLP architectures to solve real-world problems such as breast cancer diagnosis, active sonar target classification, wheat classification, and flower classification. The CSA is used to find the optimal weights and biases that will significantly increase the classification accuracy of the MLP. The performance of our proposed approach is compared with other popular training methods: genetic algorithm (GA), ant colony optimization (ACO), particle swarm optimization (PSO), Harris hawks optimization (HHO), moth-flame optimization (MFO), flower pollination algorithm (FPA), and backpropagation (BP). The comparison is benchmarked using five classification datasets: Iris Flower, Sonar, Wheat Seeds, Breast Cancer Wisconsin, and Haberman’s Survival. Comparative study results illustrate the improvements in MLP performance gained by using CSA over other training methods, and hence it can be considered a competitive approach to training MLP networks when solving real-world applications in various disciplines.

Improving surface melt estimation over the Antarctic Ice Sheet using deep learning: a proof of concept over the Larsen Ice Shelf

Abstract. Accurately estimating the surface melt volume of the Antarctic Ice Sheet is challenging and has hitherto relied on climate modeling or observations from satellite remote sensing. Each of these methods has its limitations, especially in regions with high surface melt. This study aims to demonstrate the potential of improving surface melt simulations with a regional climate model by deploying a deep learning model. A deep-learning-based framework has been developed to correct surface melt from the regional atmospheric climate model version 2.3p2 (RACMO2), using meteorological observations from automatic weather stations (AWSs) and surface albedo from satellite imagery. The framework includes three steps: (1) training a deep multilayer perceptron (MLP) model using AWS observations, (2) correcting Moderate Resolution Imaging Spectroradiometer (MODIS) albedo observations, and (3) using these two to correct the RACMO2 surface melt simulations. Using observations from three AWSs at the Larsen B and C ice shelves, Antarctica, cross-validation shows a high accuracy (root-mean-square error of 0.95 mm w.e. d−1, mean absolute error of 0.42 mm w.e. d−1, and a coefficient of determination of 0.95). Moreover, the deep MLP model outperforms conventional machine learning models and a shallow MLP model. When applying the trained deep MLP model over the entire Larsen Ice Shelf, the resulting corrected RACMO2 surface melt shows a better correlation with the AWS observations for two out of three AWSs. However, for one location (AWS 18), the deep MLP model does not show improved agreement with AWS observations; this is likely because surface melt is largely driven by factors (e.g., air temperature, topography, katabatic wind) other than albedo within the corresponding coarse-resolution model pixels. Our study demonstrates the opportunity to improve surface melt simulations using deep learning combined with satellite albedo observations. However, more work is required to refine the method, especially for complicated and heterogeneous terrains.

Training of the feed-forward artificial neural networks using butterfly optimization algorithm

Artificial Neural Network (ANN) learns from inputs and outputs. The values of the weights and biases in ANN are updated according to inputs and outputs. Researchers have proposed algorithms to train Multi-Layer Perceptron (MLP). However, classical techniques often face problems in solving this optimization problem. They tend to need large amounts of computing time, large amounts of memory. More importantly, they get stuck within the local optimum and produce poor-quality solutions. To overcome these difficulties, meta-heuristic algorithms have been used to train MLP. In this article, the Butterfly Optimization Algorithm (BOA) which was designed by modeling the behaviors of butterflies was used for the first time to train the multi-layer perceptron. The developed algorithm was named BOA-MLP where the BOA algorithm optimized the values of the weights and biases in the MLP. The success of the BOA-MLP algorithm was tested on five data sets (iris, breast cancer, heart, balloon and xor) which are frequently used in the literature. In the experiments, the BOA-MLP algorithm was compared with the BAT-MLP, SMS-MLP and BP algorithms. The average and standard deviation of the mean squared error, the average classification accuracy, the sensitivity, the specificity, the precision and the F1-score were used as the performance metrics. According to the experimental results, it is seen that the BOA-MLP algorithm surpasses the BAT-MLP, SMS-MLP and BP algorithms on all data sets and shows superior success.

Using Fitness Dependent Optimizer for Training Multi-layer Perceptron

This study presents a novel training algorithm depending upon the recently proposed Fitness Dependent Optimizer (FDO). The stability of this algorithm has been verified and performance-proofed in both the exploration and exploitation stages using some standard measurements. This influenced our target to gauge the performance of the algorithm in training multilayer perceptron neural networks (MLP). This study combines FDO with MLP (codename FDO-MLP) for optimizing weights and biases to predict outcomes of students. This study can improve the learning system in terms of the educational background of students besides increasing their achievements. The experimental results of this approach are affirmed by comparing with the Back-Propagation algorithm (BP) and some evolutionary models such as FDO with cascade MLP (FDO-CMLP), Grey Wolf Optimizer (GWO) combined with MLP (GWO-MLP), modified GWO combined with MLP (MGWO-MLP), GWO with cascade MLP (GWO-CMLP), and modified GWO with cascade MLP (MGWO-CMLP). The qualitative and quantitative results prove that the proposed approach using FDO as a trainer can outperform the other approaches using different trainers on the dataset in terms of convergence speed and local optima avoidance. The proposed FDO-MLP approach classifies with a rate of 0.97.

Hybrid Multilayer Perceptron Neural Network with Grey Wolf Optimization for Predicting Stock Market Index

Stock market forecasting is a challenging task for investors and researchers in the financial market due to highly noisy, nonparametric, volatile, complex, non-linear, dynamic and chaotic nature of stock price time series. With the development of computationally intelligent method, it is possible to predict stock price time series more accurately. Artificial neural networks (ANNs) are one of the most promising biologically inspired techniques. ANNs have been widely used to make predictions in various research. The performance of ANNs is very dependent on the learning technique utilized to train the weight and bias vectors. The proposed study aims to predict daily Tehran Exchange Dividend Price Index (TEDPIX) via the hybrid multilayer perceptron (MLP) neural networks and metaheuristic algorithms which consist of genetic algorithm (GA), particle swarm optimization (PSO), black hole (BH), grasshopper optimization algorithm (GOA) and grey wolf optimization (GWO). We have extracted 18 technical indicators based on the daily TEDPIX as input parameters. Therefore, the experimental result shows that grey wolf optimization has superior performance to train MLPs for predicting the stock market in metaheuristic-based.

Training of the Artificial Neural Networks using Crow Search Algorithm

Artificial Neural Networks are a method frequently used in problem-solving today. In the past, it has been used in many areas such as classification, pattern recognition and image processing. The most important and demanding part of Artificial Neural Networks is the training process of the network. The main challenge in network training is the process of determining the optimum connection weights and bias values for the network. In the literature, many algorithms have been proposed for training Artificial Neural Networks. This article proposed a new hybrid algorithm called CSA-MLP for training Artificial Neural Networks using the Crow Search Algorithm. Crow Search Algorithm is a population-based meta-heuristic optimization algorithm, inspired by the behavior of crows to store their surplus nutrients and take them back from the storage area when needed. Crow Search Algorithm has been proposed to solve different optimization problems in terms of its simplicity with two different adjustable parameters (flight length and awareness probability), obtaining an effective convergence rate in a short time and having a faster technique compared to algorithms frequently used in engineering problems with different constraints and functions. In the experiments, five classification datasets (xor, balloon, iris, breast cancer, heart) were used. The CSA-MLP algorithm was compared with the SMS-MLP in terms of the mean squared error, classification rate, the statistical metrics (sensitivity, specificity, precision, f1-score) and the convergence graph. Furthermore, the proposed CSA-MLP algorithm was compared with seven algorithms in literature in terms of best classification accuracy. The experimental results show that the Crow Search Algorithm is a reliable approach in training Multi-Layer Perceptron. CSA-MLP achieved better results than SMS-MLP and other algorithms.

Battle royale optimizer for training multi-layer perceptron

Battle royale optimizer for training multi-layer perceptron

Bispectral Analysis of Heart Rate Variability to Characterize and Help Diagnose Pediatric Sleep Apnea.

Pediatric obstructive sleep apnea (OSA) is a breathing disorder that alters heart rate variability (HRV) dynamics during sleep. HRV in children is commonly assessed through conventional spectral analysis. However, bispectral analysis provides both linearity and stationarity information and has not been applied to the assessment of HRV in pediatric OSA. Here, this work aimed to assess HRV using bispectral analysis in children with OSA for signal characterization and diagnostic purposes in two large pediatric databases (0–13 years). The first database (training set) was composed of 981 overnight ECG recordings obtained during polysomnography. The second database (test set) was a subset of the Childhood Adenotonsillectomy Trial database (757 children). We characterized three bispectral regions based on the classic HRV frequency ranges (very low frequency: 0–0.04 Hz; low frequency: 0.04–0.15 Hz; and high frequency: 0.15–0.40 Hz), as well as three OSA-specific frequency ranges obtained in recent studies (BW1: 0.001–0.005 Hz; BW2: 0.028–0.074 Hz; BWRes: a subject-adaptive respiratory region). In each region, up to 14 bispectral features were computed. The fast correlation-based filter was applied to the features obtained from the classic and OSA-specific regions, showing complementary information regarding OSA alterations in HRV. This information was then used to train multi-layer perceptron (MLP) neural networks aimed at automatically detecting pediatric OSA using three clinically defined severity classifiers. Both classic and OSA-specific MLP models showed high and similar accuracy (Acc) and areas under the receiver operating characteristic curve (AUCs) for moderate (classic regions: Acc = 81.0%, AUC = 0.774; OSA-specific regions: Acc = 81.0%, AUC = 0.791) and severe (classic regions: Acc = 91.7%, AUC = 0.847; OSA-specific regions: Acc = 89.3%, AUC = 0.841) OSA levels. Thus, the current findings highlight the usefulness of bispectral analysis on HRV to characterize and diagnose pediatric OSA.

Diagnosing various failures of lithium-ion batteries using artificial neural network enhanced by likelihood mapping

A simple but effective framework of diagnosing various failures of lithium-ion batteries (LIBs) equipped inside portable smart devices is proposed. In this framework, to detect failure event of batteries from partial charging curves obtained under adaptively varying charging scenarios, which are frequently the only dataset available from lithium ion battery packs equipped inside portable smart devices, partial charging data are mapped into characteristic statistical entity which we refer to as likelihood vector. Likelihood vectors are calculated by referring to probability distribution functions (PDFs) of voltage and current obtained from the experiments simulating various degradation/abuse conditions for LIBs. Compared to the brute-force training method using partial charging curves to train multi-layer perceptron (MLP) classifier models, training assisted by likelihood vectors leads to improvements in test set classification accuracy by 26 – 85% according to the size of neural networks. As a result, the optimized classification model achieves 99.8% precision for healthy data classification and 97.3% of average precision for abused data classification, reaching overall classification accuracy of 97.8%. Furthermore, by monitoring the failure index calculated from the cumulated list of detections made, it is experimentally demonstrated that the thermal runaway and resultant fatal explosion event of lithium pouch cell under operando dent test can be predicted before the event actually occurs.

Prediction of Food Production Using Machine Learning Algorithms of Multilayer Perceptron and ANFIS

Advancing models for accurate estimation of food production is essential for policymaking and managing national plans of action for food security. This research proposes two machine learning models for the prediction of food production. The adaptive network-based fuzzy inference system (ANFIS) and multilayer perceptron (MLP) methods are used to advance the prediction models. In the present study, two variables of livestock production and agricultural production were considered as the source of food production. Three variables were used to evaluate livestock production, namely livestock yield, live animals, and animal slaughtered, and two variables were used to assess agricultural production, namely agricultural production yields and losses. Iran was selected as the case study of the current study. Therefore, time-series data related to livestock and agricultural productions in Iran from 1961 to 2017 have been collected from the FAOSTAT database. First, 70% of this data was used to train ANFIS and MLP, and the remaining 30% of the data was used to test the models. The results disclosed that the ANFIS model with generalized bell-shaped (Gbell) built-in membership functions has the lowest error level in predicting food production. The findings of this study provide a suitable tool for policymakers who can use this model and predict the future of food production to provide a proper plan for the future of food security and food supply for the next generations.

An enhanced chimp optimization algorithm for continuous optimization domains

Chimp optimization algorithm (ChOA) is a recently proposed metaheuristic. Interestingly, it simulates the social status relationship and hunting behavior of chimps. Due to the more flexible and complex application fields, researchers have higher requirements for native algorithms. In this paper, an enhanced chimp optimization algorithm (EChOA) is proposed to improve the accuracy of solutions. First, the highly disruptive polynomial mutation is used to initialize the population, which provides the foundation for global search. Next, Spearman’s rank correlation coefficient of the chimps with the lowest social status is calculated with respect to the leader chimp. To reduce the probability of falling into the local optimum, the beetle antennae operator is used to improve the less fit chimps while gaining visual capability. Three strategies enhance the exploration and exploitation of the native algorithm. To verify the function optimization performance, EChOA is comprehensively analyzed on 12 classical benchmark functions and 15 CEC2017 benchmark functions. Besides, the practicability of EChOA is also highlighted by three engineering design problems and training multilayer perceptron. Compared with ChOA and five state-of-the-art algorithms, the statistical results show that EChOA has strong competitive capabilities and promising prospects.

Deep learning in forensic gunshot wound interpretation\u2014a proof-of-concept study

While the applications of deep learning are considered revolutionary within several medical specialties, forensic applications have been scarce despite the visual nature of the field. For example, a forensic pathologist may benefit from deep learning-based tools in gunshot wound interpretation. This proof-of-concept study aimed to test the hypothesis that trained neural network architectures have potential to predict shooting distance class on the basis of a simple photograph of the gunshot wound. A dataset of 204 gunshot wound images (60 negative controls, 50 contact shots, 49 close-range shots, and 45 distant shots) was constructed on the basis of nineteen piglet carcasses fired with a .22 Long Rifle pistol. The dataset was used to train, validate, and test the ability of neural net architectures to correctly classify images on the basis of shooting distance. Deep learning was performed using the AIDeveloper open-source software. Of the explored neural network architectures, a trained multilayer perceptron based model (MLP_24_16_24) reached the highest testing accuracy of 98%. Of the testing set, the trained model was able to correctly classify all negative controls, contact shots, and close-range shots, whereas one distant shot was misclassified. Our study clearly demonstrated that in the future, forensic pathologists may benefit from deep learning-based tools in gunshot wound interpretation. With these data, we seek to provide an initial impetus for larger-scale research on deep learning approaches in forensic wound interpretation.

Efficient activation functions for embedded inference engines

The importance of the choice of the activation function for training and inferencing in machine learning cannot be overemphasized. Activation functions can influence network training convergence, performance accuracy, and can make training and inference stages computationally expensive. We introduce a family of square-based activation functions for embedded devices that consume only one instruction cycle with the potential of being resource efficient when constructed in silicon. We show that the proposed family are computationally efficient when compared with exponential-based non-linearities. The family includes functions for deep neural network architectures, support vector machines, recurrent neural networks, and others. We demonstrate the universal ability of the square-based family of activation functions on a variety of neural network architectures. We analyze the hidden representations of our trained multilayer perceptron network in an attempt to explain the performance gains, and speed-up observed when using square non-linearities. Speed-up was recorded with the quadratic-based kernel transformation on support vector machines. We record higher performance accuracy for recurrent neural network architectures and logistic regression using this family. Speed up was also recorded for implementation on Intel CPU and ARM processors. This family will find particular importance in low-end hardware devices with limited hardware capabilities.

Automatic detection of annual rings and pith location along Norway spruce timber boards using conditional adversarial networks

In the woodworking industry, detection of annual rings and location of pith in relation to timber board cross sections, and how these properties vary in the longitudinal direction of boards, is relevant for many purposes such as assessment of shape stability and prediction of mechanical properties of timber. The current work aims at developing a fast, accurate and operationally simple deep learning-based algorithm for automatic detection of surface growth rings and pith location along knot-free clear wood sections of Norway spruce boards. First, individual surface growth rings that are visible along the four longitudinal sides of the scanned boards are detected using trained conditional generative adversarial networks (cGANs). Then, pith locations are determined, on the basis of the detected growth rings, by using a trained multilayer perceptron (MLP) artificial neural network. The proposed algorithm was solely based on raw images of board surfaces obtained from optical scanning and applied to a total of 104 Norway spruce boards with nominal dimensions of 45times 145times 4500,hbox {mm}^{3}. The results show that optical scanners and the proposed automatic method allow for accurate and fast detection of individual surface growth rings and pith location along boards. For boards with the pith located within the cross section, median errors of 1.4 mm and 2.9 mm, in the x- and y-direction, respectively, were obtained. For a sample of boards with the pith located outside the board cross section in most positions along the board, the median discrepancy between automatically estimated and manually determined pith locations was 3.9 mm and 5.4 mm in the x- and y-direction, respectively.

AGWO: Advanced GWO in multi-layer perception optimization

The Multi-Layer Perceptron (MLP) has been applied into many real-world problems as one of the most extensively used Neural Networks (NNs). It often suffers from local stagnation and premature convergence problems when treating the specific datasets. This paper proposes an Advanced Grey Wolf Optimization algorithm (AGWO) with elastic, circling and attacking mechanisms to alleviate those problems. The performance of the proposed AGWO is tested on the IEEE CEC 2014. Meanwhile, the efficiency of AGWO and five noted heuristic algorithms in training MLP are investigated by 7 classification datasets and 3 function approximation datasets. The experimental results show that AGWO is superior to other algorithms in terms of local stagnation and premature convergence. Simultaneously, training MLP by means of AGWO is clearly superior to the other heuristic algorithms in local optimum avoidance and computational accuracy.