Multilayer Perceptron Training Research Articles

A New Fusion of ASO with SA Algorithm and Its Applications to MLP Training and DC Motor Speed Control

An improved version of atom search optimization (ASO) algorithm is proposed in this paper. The search capability of ASO was improved by using simulated annealing (SA) algorithm as an embedded part of it. The proposed hybrid algorithm was named as hASO-SA and used for optimizing nonlinear and linearized problems such as training multilayer perceptron (MLP) and proportional-integral-derivative controller design for DC motor speed regulation as well as testing benchmark functions of unimodal, multimodal, hybrid and composition types. The obtained results on classical and CEC2014 benchmark functions were compared with other metaheuristic algorithms, including two other SA-based hybrid versions, which showed the greater capability of the proposed approach. In addition, nonparametric statistical test was performed for further verification of the superior performance of hASO-SA. In terms of MLP training, several datasets were used and the obtained results were compared with respective competitive algorithms. The results clearly indicated the performance of the proposed algorithm to be better. For the case of controller design, the performance evaluation was performed by comparing it with the recent studies adopting the same controller parameters and limits as well as objective function. The transient, frequency and robustness analysis demonstrated the superior ability of the proposed approach. In brief, the comparative analyses indicated the proposed algorithm to be successful for optimization problems with different nature.

Multi-Criteria Decision Support System for Lung Cancer Prediction

Lung cancer is one of the most common deadly malignant tumours, with the most rapid morbidity and death worldwide. Cancer risk prediction is a challenging and complex task in the field of healthcare. Many studies have been carried out by researchers to analyse and establish lung cancer symptoms and factors. However, further improvements are vital and required to be conducted in order to overcome the persistent challenges. In this study, a multi-criteria decision support system for lung cancer risk prediction based on a web-based survey data has been presented and realised. The proposed framework aims to incorporate the powerful of analytical hierarchy process (AHP) with artificial neural network for constituting lung cancer prediction model. The multiple criteria decision-making strategy (AHP) assigns a weight to each individual cancer symptom feature from survey data. The weighted features are then used to train multi-layer perceptron artificial neural network (ANN) to build a disease prediction model. Experimental analysis and evaluation performed on 276 subjects revealed promising prediction performance of developed lung cancer prediction framework in terms of various classification metrics.

SmartSORT: an MLP-based method for tracking multiple objects in real-time

With the recent advances in the object detection research field, tracking-by-detection has become the leading paradigm adopted by multi-object tracking algorithms. By extracting different features from detected objects, those algorithms can estimate the similarities and association patterns of objects along with successive frames. However, since similarity functions applied by tracking algorithms are handcrafted, it is difficult to use them in new contexts. In this study, it is investigated the use of artificial neural networks to learning a similarity function that can be used among detections. During training, multilayer perceptron (MLP) neural networks were introduced to correct and incorrect association patterns, sampled from a pedestrian tracking data set. For such, different motion and appearance feature combinations have been explored. Finally, a trained MLP has been inserted into a multiple-object tracking framework, which has been assessed on the MOT Challenge benchmark. Throughout the experiments, the proposed tracker matched the results obtained by state-of-the-art methods by scoring a tracking accuracy of 60.4%, while running 58% faster than DeepSORT, a recent and similar method used as a baseline. After all, this work demonstrates its method can be automatically trained for different tracking contexts and it has highly competitive cost-effectiveness for online real-time tracking applications.

Analyzing the Contact Problem of a Functionally Graded Layer Resting on an Elastic Half Plane with Theory of Elasticity, Finite Element Method and Multilayer Perceptron

This paper presents a comparative study of analytical method, finite element method (FEM) and Multilayer Perceptron (MLP) for analysis of a contact problem. The problem consists of a functionally graded (FG) layer resting on a half plane and pressed with distributed load from the top. Firstly, analytical solution of the problem is obtained by using theory of elasticity and integral transform techniques. The problem is reduced a system of integral equation in which the contact pressure are unknown functions. The numerical solution of the integral equation was carried out with Gauss-Jacobi integration formulation. Secondly, finite element model of the problem is constituted using ANSYS software and the two-dimensional analysis of the problem is carried out. The results show that contact areas and the contact stresses obtained from FEM provide boundary conditions of the problem as well as analytical results. Thirdly, the contact problem has been extended based on the MLP. The MLP with three-layer was used to calculate the contact distances. Material properties and loading states were created by giving examples of different values were used at the training and test stages of MLP. Program code was rewritten in C++. As a result, average deviation values such as 0.375 and 1.465 was obtained for FEM and MLP respectively. The contact areas and contact stresses obtained from FEM and MLP are very close to results obtained from analytical method. Finally, this study provides evidence that there is a good agreement between three methods and the stiffness parameters has an important effect on the contact stresses and contact areas.

Prediction of Food Production Using Machine Learning Algorithms of Multilayer Perceptron and ANFIS

Advancing models for accurate estimation of food production is essential for policymaking and managing national plans of action for food security. This research proposes two machine learning models for the prediction of food production. The adaptive network-based fuzzy inference system (ANFIS) and multilayer perceptron (MLP) methods are used to advance the prediction models. In the present study, two variables of livestock production and agricultural production were considered as the source of food production. Three variables were used to evaluate livestock production, namely livestock yield, live animals, and animal slaughtered, and two variables were used to assess agricultural production, namely agricultural production yields and losses. Iran was selected as the case study of the current study. Therefore, time-series data related to livestock and agricultural productions in Iran from 1961 to 2017 have been collected from the FAOSTAT database. First, 70% of this data was used to train ANFIS and MLP, and the remaining 30% of the data was used to test the models. The results disclosed that the ANFIS model with Generalized bell-shaped (Gbell) built-in membership functions has the lowest error level in predicting food production. The findings of this study provide a suitable tool for policymakers who can use this model and predict the future of food production to provide a proper plan for the future of food security and food supply for the next generations.

Surface roughness prediction of FFF-fabricated workpieces by artificial neural network and Box–Behnken method

Fused Filament Fabrication (FFF) or Fused Deposition Modelling (FDM) or three-dimension (3D) printing are rapid prototyping processes for workpieces. There are many factors which have a significant effect on surface quality, including bed temperature, printing speed, and layer thickness. This empirical study was conducted to determine the relationship between the above-mentioned factors and average surface roughness (Ra). Workpieces of cylindrical shape were fabricated by an FFF system with a Polylactic acid (PLA) filament. The surface roughness was measured at five different positions on the bottom and top surface. A response surface (Box-Behnken) method was utilised to design the experiment and statistically predict the response. The total number of treatments was sixteen, while five measurements (Ra1, Ra2, Ra3, Ra4 and Ra5) were carried out for each treatment. The settings of each factor were as follows: bed temperature (80, 85, and 90 °C), printing speed (40, 80 and 120 mm/s), and layer thickness (0.10, 0.25 and 0.40 mm). The prediction equation of surface roughness was then derived from the analysis. The same set of data was also used as the inputs for a machine learning method, an artificial neural network (ANN), to construct the prediction equation of surface roughness. Rectified linear unit (ReLU) was utilised as the activation function of ANN. Two training algorithms (resilient backpropagation with weight backtracking and globally convergent resilient backpropagation) were applied to train multi-layer perceptrons. Moreover, the different number of neurons in each hidden layer was also studied and compared. Another interesting aspect of this study is that the ANN was based on a limited number of training samples. Finally, the prediction errors of each method were compared, to benchmark the prediction performance of the two methods: Box-Behnken and ANN.

Network Compression via Mixed Precision Quantization Using a Multi-Layer Perceptron for the Bit-Width Allocation

Deep Neural Networks (DNNs) are a powerful tool for solving complex tasks in many application domains. The high performance of DNNs demands significant computational resources, which might not always be available. Network quantization with mixed-precision across the layers can alleviate this high demand. However, determining layer-wise optimal bit-widths is non-trivial, as the search space is exponential. This article proposes a novel technique for allocating layer-wise bit-widths for a DNN using a multi-layer perceptron (MLP). The Kullback-Leibler (KL) divergence of the softmax outputs between the quantized and full precision network is used as the metric to quantify the quantization quality. We explore the relationship between the KL-divergence and the network size, and from our experiments observe that more aggressive quantization leads to higher divergence, and vice versa. The MLP is trained with layer-wise bit-widths as labels and their corresponding KL-divergence as the input. The MLP training set, i.e. the pairs of the layer-wise bit-widths and their corresponding KL-divergence, is collected using a Monte Carlo sampling of the exponential search space. We introduce a penalty term in the loss to ensure that the MLP learns to predict bit-widths resulting in the smallest network size. We show that the layer-wise bit-width predictions from the trained MLP result in reduced network size without degrading accuracy while achieving better or comparable results with SOTA work but with less computational overhead. Our method achieves up to 6x, 4x, 4x compression on VGG16, ResNet50, and GoogLeNet respectively, with no accuracy drop compared to the original full precision pretrained model, on the ImageNet dataset.

Neural network modeling of innovative development of JSC «Ukrtransnaft»

Methods. The following methods were used in the research: analysis and synthesis – to set the problem of neural network modeling of the level of innovative development of economic entities and the selection of its stages; methods of generalization and grouping – in the formation of a system of individual indicators to determine the level of innovative development; multilayer perceptron training method – to characterize the operation of the neural network. Results. The article considers the possibility of applying the method of forecasting the level of innovative development of enterprises of the oil transportation complex with the use of neural network modeling. The problem of neural network modeling of the level of innovative development of economic entities is set, and its six stages are defined: formation of a system of indicators of the level of innovative development, preliminary data processing, neural network formation, neural network training and result. There has been formed a system of individual indicators for determining the level of innovative development by production, personnel, financial and property components. These indicators are used to build a neural network. There are specific limitations that characterize the learning of the neural network and distinguish it from the general optimization problems: astronomical number of parameters, the need for high parallelism in learning, multicriteria of problems, the need to find an area in which the values of all minimized functions are close to minimum. The advantages and disadvantages of neural network forecasting of the level of innovative development of enterprises are determined. Novelty. An algorithm for forecasting the level of innovative development of oil transportation enterprises using neural network modeling is proposed. A system of individual indicators for determining the level of innovative development by production, personnel, financial and property components has been formed. Practical value. The results of the study have theoretical and practical value, as they allow the use of neural network modeling to predict the level of innovative development of enterprises.

Biologically-informed neural networks guide mechanistic modeling from sparse experimental data.

Biologically-informed neural networks (BINNs), an extension of physics-informed neural networks [1], are introduced and used to discover the underlying dynamics of biological systems from sparse experimental data. In the present work, BINNs are trained in a supervised learning framework to approximate in vitro cell biology assay experiments while respecting a generalized form of the governing reaction-diffusion partial differential equation (PDE). By allowing the diffusion and reaction terms to be multilayer perceptrons (MLPs), the nonlinear forms of these terms can be learned while simultaneously converging to the solution of the governing PDE. Further, the trained MLPs are used to guide the selection of biologically interpretable mechanistic forms of the PDE terms which provides new insights into the biological and physical mechanisms that govern the dynamics of the observed system. The method is evaluated on sparse real-world data from wound healing assays with varying initial cell densities [2].

Training Multilayer Perceptron with Genetic Algorithms and Particle Swarm Optimization for Modeling Stock Price Index Prediction.

Predicting stock market (SM) trends is an issue of great interest among researchers, investors and traders since the successful prediction of SMs’ direction may promise various benefits. Because of the fairly nonlinear nature of the historical data, accurate estimation of the SM direction is a rather challenging issue. The aim of this study is to present a novel machine learning (ML) model to forecast the movement of the Borsa Istanbul (BIST) 100 index. Modeling was performed by multilayer perceptron–genetic algorithms (MLP–GA) and multilayer perceptron–particle swarm optimization (MLP–PSO) in two scenarios considering Tanh (x) and the default Gaussian function as the output function. The historical financial time series data utilized in this research is from 1996 to 2020, consisting of nine technical indicators. Results are assessed using Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE) and correlation coefficient values to compare the accuracy and performance of the developed models. Based on the results, the involvement of the Tanh (x) as the output function, improved the accuracy of models compared with the default Gaussian function, significantly. MLP–PSO with population size 125, followed by MLP–GA with population size 50, provided higher accuracy for testing, reporting RMSE of 0.732583 and 0.733063, MAPE of 28.16%, 29.09% and correlation coefficient of 0.694 and 0.695, respectively. According to the results, using the hybrid ML method could successfully improve the prediction accuracy.

Combined texture analysis and machine learning in suspicious calcifications detected by mammography: Potential to avoid unnecessary stereotactical biopsies

ObjectivesTo investigate whether combined texture analysis and machine learning can distinguish malignant from benign suspicious mammographic calcifications, to find an exploratory rule-out criterion to potentially avoid unnecessary benign biopsies. MethodsMagnification views of 235 patients which underwent vacuum-assisted biopsy of suspicious calcifications (BI-RADS 4) during a two-year period were retrospectively analyzed using the texture analysis tool MaZda (Version 4.6). Microcalcifications were manually segmented and analyzed by two readers, resulting in 249 image features from gray-value histogram, gray-level co-occurrence and run-length matrices. After feature reduction with principal component analysis (PCA), a multilayer perceptron (MLP) artificial neural network was trained using histological results as the reference standard. For training and testing of this model, the dataset was split into 70 % and 30 %. ROC analysis was used to calculate diagnostic performance indices. Results226 patients (150 benign, 76 malignant) were included in the final analysis due to missing data in 9 cases. Feature selection yielded nine image features for MLP training. Area under the ROC-curve in the testing dataset (n = 54) was 0.82 (95 %-CI: 0.70−0.94) and 0.832 (95 %-CI 0.72−0.94) for both readers, respectively. A high sensitivity threshold criterion was identified in the training dataset and successfully applied to the testing dataset, demonstrating the potential to avoid 37.1–45.7 % of unnecessary biopsies at the cost of one false-negative for each reader. ConclusionCombined texture analysis and machine learning could be used for risk stratification in suspicious mammographic calcifications. At low costs in terms of false-negatives, unnecessary biopsies could be avoided.

A Learning-Based Framework for Detecting Cyber-Attacks Against Line Current Differential Relays

Technical developments in communication technology and measurement synchronization have facilitated the design of advanced protection schemes, such as Line Current Differential Relays (LCDRs). However, the superior performance of LCDRs is achieved at the expense of exposing them to cyber-threats, since cyber-induced intrusions against protective relays-which take advantage of the direct control of relays over circuit-breakers-can cause protection system mis-operations. To address this problem, this paper presents a Learning-based Framework (LBF) for detecting False Data Injection Attacks (FDIAs) and Time Synchronization Attacks (TSAs) against LCDRs, and for differentiating them from faults. In the proposed LBF, a Multi-Layer Perceptron (MLP) model is trained based on differential and super-imposed features, which are selected using the Recursive Feature Elimination method. After implementing the proposed LBF in LCDRs, when an LCDR picks up, it initially extracts the features and sends them to the trained MLP model. The LCDR trips the line if the proposed LBF confirms a fault. The performance of the proposed LBF is corroborated using the IEEE 39-bus test system. Evaluation results show that the proposed LBF (i) works independently of a system's operating point and configuration, (ii) is not considerably affected by instrumentation errors, and (iii) can accurately detect FDIAs and TSAs.

Machine learning, alignment of covariant Lyapunov vectors, and predictability in Rikitake's geomagnetic dynamo model.

In this paper, the alignment of covariant Lyapunov vectors is used to train multi-layer perceptron ensembles in order to predict the duration of regimes in chaotic time series of Rikitake's geomagnetic dynamo model. The machine learning procedure reveals the relevance of the alignment of distinct covariant Lyapunov vectors for the predictions. To train multi-layer perceptron, we use a classification procedure that associates the number of maxima (or minima) inside regimes of motion with the duration of the corresponding regime. Remarkably accurate predictions are obtained, even for the longest regimes whose duration times are around 17.5 Lyapunov times. We also found long duration regimes with a distinctive statistical behavior, namely, the longest regimes are more likely to occur, a quite unusual behavior. In fact, we observed a largest regime above which no regimes were observed.

An efficient equilibrium optimizer with mutation strategy for numerical optimization

To alleviate the shortcomings of the standard Equilibrium Optimizer, a new improved algorithm called Modified Equilibrium Optimizer is proposed in this work. This algorithm utilizes the Gaussian mutation and an additional exploratory search mechanism based on the concept of population division and reconstruction. The population in each iteration of the proposed algorithm is constructed using these mechanisms and standard search procedure of the Equilibrium Optimizer. These strategies attempt to maintain the diversity of solutions during the search, so that the tendency of stagnation towards the sub-optimal solutions can be avoided and the convergence rate can be boosted to obtain more accurate optimal solutions. To validate and analyze the performance of the Modified Equilibrium Optimizer, a collection of 33 benchmark problems and four engineering design problems are adopted. Later, in the paper, the Modified Equilibrium Optimizer has been used to train multilayer perceptrons. The experimental results and comparison based on several metrics such as statistical analysis, scalability test, diversity analysis, performance index analysis and convergence analysis demonstrate that the proposed algorithm can be considered a better metaheuristic optimization approach than other compared algorithms.

Training multi-layer perceptron with artificial algae algorithm

Artificial Neural Networks are commonly used to solve problems in many areas, such as classification, pattern recognition, and image processing. The most challenging and critical phase of an Artificial Neural Networks is related with its training process. The main challenge in the training process is finding optimal network parameters (i.e. weight and biase). For this purpose, numerous heuristic algorithms have been used. One of them is Artificial Algae Algorithm, which has a nature-inspired metaheuristic optimization algorithm. This algorithm is capable of successfully solving a wide variety of numerical optimization problems. In this study, Artificial Algae Algorithm is proposed for training Artificial Neural Network. Ten classification datasets with different degrees of difficulty from the UCI database repository were used to compare the proposed method performance with six well known swarm-based optimization and backpropagation algorithms. The results of the study show that Artificial Algae Algorithm is a reliable approach for training Artificial Neural Networks.

A hybrid constriction coefficient-based particle swarm optimization and gravitational search algorithm for training multi-layer perceptron

PurposeIn this paper, a newly proposed hybridization algorithm namely constriction coefficient-based particle swarm optimization and gravitational search algorithm (CPSOGSA) has been employed for training MLP to overcome sensitivity to initialization, premature convergence, and stagnation in local optima problems of MLP.Design/methodology/approachIn this study, the exploration of the search space is carried out by gravitational search algorithm (GSA) and optimization of candidate solutions, i.e. exploitation is performed by particle swarm optimization (PSO). For training the multi-layer perceptron (MLP), CPSOGSA uses sigmoid fitness function for finding the proper combination of connection weights and neural biases to minimize the error. Secondly, a matrix encoding strategy is utilized for providing one to one correspondence between weights and biases of MLP and agents of CPSOGSA.FindingsThe experimental findings convey that CPSOGSA is a better MLP trainer as compared to other stochastic algorithms because it provides superior results in terms of resolving stagnation in local optima and convergence speed problems. Besides, it gives the best results for breast cancer, heart, sine function and sigmoid function datasets as compared to other participating algorithms. Moreover, CPSOGSA also provides very competitive results for other datasets.Originality/valueThe CPSOGSA performed effectively in overcoming stagnation in local optima problem and increasing the overall convergence speed of MLP. Basically, CPSOGSA is a hybrid optimization algorithm which has powerful characteristics of global exploration capability and high local exploitation power. In the research literature, a little work is available where CPSO and GSA have been utilized for training MLP. The only related research paper was given by Mirjalili et al., in 2012. They have used standard PSO and GSA for training simple FNNs. However, the work employed only three datasets and used the MSE performance metric for evaluating the efficiency of the algorithms. In this paper, eight different standard datasets and five performance metrics have been utilized for investigating the efficiency of CPSOGSA in training MLPs. In addition, a non-parametric pair-wise statistical test namely the Wilcoxon rank-sum test has been carried out at a 5% significance level to statistically validate the simulation results. Besides, eight state-of-the-art meta-heuristic algorithms were employed for comparative analysis of the experimental results to further raise the authenticity of the experimental setup.

A Forecasting with Multilayer Perceptron Algorithm the Occupancy Rate of Accommodation Establishments in Turkey

Artificial neural networks (ANN), which is one of the applications of artificial intelligence, is the information processing technology that analyzes the existing data by mimicking the working structure of the human brain and creates new information with different learning algorithms. In recent years, ANN have become popular in scientific and business fields. In the hotel industry, researchers have recently focused on the classification of tourist segments of neural networks and predicting visitor behavior. However, it was requested to include ANN in the hotel occupancy rate prediction. In this paper, forecasted occupancy. rate of the hotel in Turkey by using ANN's a class that Multilayer Perceptron (MLP) algorithm. This study were used “Accommodation Statistics” of monthly data between 2000-2018 years obtained from the Ministry of Culture and Tourism. 3 input values and 1 output value were used in the training of MLP developed for hotel occupancy rate estimation. As a result of the study, a prediction that occupancy rate close to the actual occupancy rate was obtained. And it has been found to have low error rates. The success rate of the algorithm has been seen that be per cent 91.85%. In future studies, it is seen that occupancy rates of hotels operating within a certain region or province limits, tourist spending in hotels, number of overnight stays and average lengths of stay can be estimated by using more variables with ANN.

Prediction of the Influential Factors on Eating Behaviors: A Hybrid Model of Structural Equation Modelling-Artificial Neural Networks.

The importance of eating behavior risk factors in the primary prevention of obesity has been established. Researchers mostly use the linear model to determine associations among these risk factors. However, in reality, the presence of nonlinearity among these factors causes a bias in the prediction models. The aim of this study was to explore the potential of a hybrid model to predict the eating behaviors. The hybrid model of structural equation modelling (SEM) and artificial neural networks (ANN) was applied to evaluate the prediction model. The SEM analysis was used to check the relationship of the emotional eating scale (EES), body shape concern (BSC), and body appreciation scale (BAS) and their effect on different categories of eating behavior patterns (EBP). In the second step, the input and output required for ANN analysis were obtained from SEM analysis and were applied in the neural network model. 340 university students participated in this study. The hybrid model (SEM-ANN) was conducted using multilayer perceptron (MLP) with feed-forward network topology. Moreover, Levenberg–Marquardt, which is a supervised learning model, was applied as a learning method for MLP training. The tangent/sigmoid function was used for the input layer, while the linear function was applied for the output layer. The coefficient of determination (R2) and mean square error (MSE) were calculated. Using the hybrid model, the optimal network happened at MLP 3-17-8. It was proved that the hybrid model was superior to SEM methods because the R2 of the model was increased by 27%, while the MSE was decreased by 9.6%. Moreover, it was found that BSC, BAS, and EES significantly affected healthy and unhealthy eating behavior patterns. Thus, a hybrid approach could be suggested as a significant methodological contribution from a machine learning standpoint, and it can be implemented as software to predict models with the highest accuracy.

Classification strategies in machine learning techniques predicting regime changes and durations in the Lorenz system

In this paper, we use machine learning strategies aiming to predict chaotic time series obtained from the Lorenz system. Such strategies prove to be successful in predicting the evolution of dynamical variables over a short period of time. Transitions between the regimes and their duration can be predicted with great accuracy by means of counting and classification strategies, for which we train multi-layer perceptron ensembles. Even for the longest regimes the occurrences and duration can be predicted. We also show the use of an echo state network to generate data of the time series with an accuracy of up to a few hundreds time steps. The ability of the classification technique to predict the regime duration of more than 11 oscillations corresponds to around 10 Lyapunov times.

A modified Sine Cosine Algorithm with novel transition parameter and mutation operator for global optimization

Inspired by the mathematical characteristics of sine and cosine trigonometric functions, the Sine Cosine Algorithm (SCA) has shown competitive performance among other meta-heuristic algorithms. However, despite its sufficient global search ability, its low exploitation ability and immature balance between exploitation and exploration remain weaknesses. In order to improve Sine Cosine Algorithm (SCA), this paper presents a modified version of the SCA called MSCA. Firstly, a non-linear transition rule is introduced instead of a linear transition to provide comparatively better transition from the exploration to exploitation. Secondly, the classical search equation of the SCA is modified by introducing the leading guidance based on the elite candidate solution. When the above proposed modified search mechanism fails to provide a better solution, in addition, a mutation operator is used to generate a new position to avoid the situation of getting trapped in locally optimal solutions during the search. Thus, the MSCA effectively maximizes the advantages of proposed strategies in maintaining a comparatively better balance of exploration and exploitation as compared to the classical SCA. The validity of the MSCA is tested on a set of 33 benchmark optimization problems and employed for training multilayer perceptrons. The numerical results and comparisons among several algorithms show the enhanced search efficiency of the MSCA.