Feedforward Neural Network Method Research Articles

A numerical verification method for multi-class feed-forward neural networks

The use of neural networks in embedded systems is becoming increasingly common, but these systems often operate in safety–critical environments, where a failure or incorrect output can have serious consequences. Therefore, it is essential to verify the expected operation of neural networks before deploying them in such settings. In this publication, we present a novel approach for verifying the correctness of these networks using a nonlinear equation system under the assumption of closed-form activation functions. Our method is able to accurately predict the output of the network for given specification intervals, providing a valuable tool for ensuring the reliability and safety of neural networks in embedded systems.

Frontiers of thermobarometry: GAIA, a novel Deep Learning-based tool for volcano plumbing systems

The anatomy of the plumbing system of active volcanoes is fundamental to understand how magma is stored and channeled to the surface. Reliable geothermobarometric estimates are, therefore, critical to assess the depths and temperatures of the complex system of magmatic reservoirs that form a volcano apparatus. Here, we developed a novel Machine Learning approach (named GAIA, Geo Artificial Intelligence thermobArometry) based upon Feedforward Neural Networks to estimate P-T conditions of magma (clinopyroxene) storage and migration within the crust. Our Feedforward Neural Network method applied to clinopyroxene compositions yields better uncertainties (Root-Mean-Square Error and R2 score) than previous Machine Learning methods and set the basis for a novel generation of reliable geothermobarometers, which extends beyond the paradigm associated to crystal-liquid equilibrium. Also, the bootstrap procedure, inherent to the Feedforward Neural Network architecture, permits to perform a rigorous assessment of the P-T uncertainty associated to each clinopyroxene composition, as opposed to the Root-Mean-Square Error representing the P-T uncertainty of whole set of clinopyroxene compositions.As a test, we applied GAIA to assess P-T conditions of five Italian volcanoes (Somma-Vesuvius, Campi Flegrei, Etna, Stromboli, Volcano), which are among the most dangerous volcanic centres in Europe. The results on the depths of the plumbing systems are in excellent agreement with those obtained with independent geophysical and geodetic surveys, and provide further evidence to current models of volcano plumbing systems consisting of physically-separated reservoirs interconnected by a network of conduits channelling magma en route to the surface. The results on the magma (clinopyroxene crystallization) temperatures are also in agreement with other estimates, albeit obtained considering - mainly but not only - thermodynamically-based clinopyroxene-liquid geothermometers.GAIA can set robust estimates of magma storage, segregation, and ascent conditions within the plumbing system of active volcanoes, helping to unravel P-T variations, if any, during their eruptive history and providing robust clues to volcanic hazard assessment.

Klasifikasi Ujaran Kebencian Menggunakan Metode FeedForward Neural Network (IndoBERT)

Everyone in Indonesia has freedom of speech, both in real life and on social media. However, freedom of speech carried out without filtering can lead to hate speech. Hate speech is a form of discrimination directed against individuals or groups of individuals based on race, religion, gender, sexual orientation, or other identities. Hate speech can harm other parties which as a result can trigger conflict, violence, and can even cost a person's life. Therefore, it is important to be able to identify and manage this hate speech effectively. One way to manage hate speech on social media is to classify it. In this study, a web-based application was created that can classify a sentence to determine whether the sentence is hate speech or a normal sentence. The model created for classification uses the feedforward neural network method with IndoBERT. Based on the test results, the model created using the feedforward neural network method with IndoBERT provides the best accuracy of 89.52%.

A Novel Model Prediction and Migration Method for Multi-Mode Nonlinear Time-Delay Processes

Most industrial processes have nonlinear and time-delay characteristics leading to difficulty in prediction modeling. In addition, the working conditions of most industrial processes are complex, which results in multiple modes. Testing and modeling for each mode is a waste of time and resources. Therefore, it is urgent to complete model migration between different modes. In this work, a new prediction model, a nonlinear autoregressive model with exogenous inputs and back propagation neural network (NARX-BP), is proposed for the nonlinear and time-delay processes, where the input data order of the model is determined by the feedforward neural network (FNN) method, and the nonlinear relation is realized by the BP neural network. For the multi-mode characteristic, a new migration optimization algorithm, input–output slope/bias correction and differential evolution (IOSBC-DE), is provided for using a small amount of data under a new mode to correct the slope and bias of the relationship between the input and output variables through DE. The modeling and migration methods are applied to a wind tunnel system, and the simulation result shows the effectiveness of the proposed method.

TwIdw—A Novel Method for Feature Extraction from Unstructured Texts

This research proposes a novel technique for fake news classification using natural language processing (NLP) methods. The proposed technique, TwIdw (Term weight–inverse document weight), is used for feature extraction and is based on TfIdf, with the term frequencies replaced by the depth of the words in documents. The effectiveness of the TwIdw technique is compared to another feature extraction method—basic TfIdf. Classification models were created using the random forest and feedforward neural networks, and within those, three different datasets were used. The feedforward neural network method with the KaiDMML dataset showed an increase in accuracy of up to 3.9%. The random forest method with TwIdw was not as successful as the neural network method and only showed an increase in accuracy with the KaiDMML dataset (1%). The feedforward neural network, on the other hand, showed an increase in accuracy with the TwIdw technique for all datasets. Precision and recall measures also confirmed good results, particularly for the neural network method. The TwIdw technique has the potential to be used in various NLP applications, including fake news classification and other NLP classification problems.

Comparison of Feedforward Neural Network and Classical Statistics Methods: Application in Finance

The flexibility and elevated accurateness of the statistical machine learning method makes this method widely applied in various fields. One of the statistical machine learning methods is the neural network, which can be used for data analysis. The great performance of the neural network method can be used in the field of finance. In this study, the neural network method was used to predict Non-Performing Loans (NPL) data and forecast credit receivables. In the NPL prediction, the banks used are State-Owned Banks, Regional Government Banks, and National Private Banks with a main capital of more than 6 trillion rupiahs in March 2021, i.e. 26 banks with the period of March 2018 until March 2021. In predicting NPL, a moving window scheme involves several different periods. In the forecast of the number of credit receivables, the data used is the number of financing receivables. The period from November 2012 to December 2020 is used as training data, while data for the period from January to June 2021 is used as testing data. The results of the analysis show that the neural network for NPL prediction and credit receivables forecasting shows better performance compared to classical methods such as multiple linear regression and ARIMA. A comparison of methods for banking NPLs prediction is based on the RMSE data testing values, while forecasting credit receivable is based on RMSE, MAE, and MAPE data testing values.

The soft computing FFNN method for adjusting heteroscedasticity on the time series model of currency exchange rate

IntroductionTime series models on financial data often have problems with the stationary assumption of variance on the residuals. It is well known as the heteroscedasticity effect. The heteroscedasticity is represented by a nonconstant value that varies over time.MethodsThe heteroscedasticity effect contained in the basic classical time series model of Autoregressive Integrated Moving Average (ARIMA) can adjust its residuals as the variance model by using Generalized Autoregressive Conditional Heteroscedasticity (GARCH). In improving the model accuracy and overcoming the heteroscedasticity problems, it is proposed a combination model of ARIMA and Feed-Forward Neural Network (FFNN), namely ARIMA-FFNN. The model is built by applying the soft computing method of FFNN to replace the variance model. This soft computing approach is one of the numerical methods that can not be only applied in the theoretical subject but also in the data processing.ResultsIn this research, the accuracy of the time series model using the case study of the exchange rate United States dollar-Indonesia rupiah with a monthly period from January 2001 to May 2021 shows that the best accuracy of the possible models is the model of ARIMA-FFNN, which applies soft computing to obtain the optimal fitted parameters precisely.DiscussionThis result indicates that the ARIMA-FFNN model is better used to approach this exchange rate than the rest model of ARIMA-GARCH and ARIMA-GARCH-FFNN.

Identification of olive leaf disease through optimized deep learning approach

The production of olives in Saudi Arabia, which accounts for around 6% of worldwide output, is regarded as one of the best in the world. Because olive trees are rain-fed and produced using conventional methods, yields vary greatly each year, which is made worse by viral illnesses and climate change. Therefore, it is necessary to identify plant illnesses early on. Farmers diagnose plant illnesses using conventional visual assessment or laboratory analysis. Diagnosing illnesses affecting olive leaves has been improved with deep learning (DL). To identify and categorize plant illnesses, this research introduces an Optimized Artificial Neural Network (ANN) that analyses the plant's leaf. Data is first integrated for preprocessing, relevant features are extracted, and the Whale Optimization Algorithm (WOA) is used to select necessary features. Then the data is classified using ANN. The ANN classification approach utilizes the feed-forward neural network method (FFNN). ANN is a highly adaptable technology being utilized widely to address various problems. This study applies categorization to exclude possibilities throughout each stage, improving prediction accuracy. Compared to the current model employed for plant disease detection, the suggested model showed a considerable performance increase in Precision, Recall, Accuracy, and F1-measure.

Hybrid Metaheuristic Model for Optimal Economic Load Dispatch in Renewable Hybrid Energy System

Hybrid generating systems in power networks have emerged as a result of the rapid growth of renewable infrastructure and widespread support for green energy. One of the most significant problems in designing and operating an electric power generation system is the efficient scheduling of all power generation facilities to meet the rising power demand. Economic load dispatch (ELD) is a generic procedure in the electrical power system, and the ELD in power system problems involves scheduling the power generating units to reduce cost and satisfy system constraints. Metaheuristic algorithms are gaining popularity for solving constrained ELD issues because of their larger global solution capacity, flexibility, and derivative-free construction. In this research, the ELD problem of integrated renewable resources is solved using a unique solution model based on hybrid optimization. Furthermore, this work considers multiobjectives such as total wind generation cost, total cost function of thermal units, and penalty cost function. The hybrid optimization model optimizes the power generation of thermal power plants within the maximum and minimum limitations. Additionally, the turbines are selected optimally by the hybrid optimization model to ensure the power generation of wind turbines based on the demands. The proposed hybrid optimization is a combination of particle swarm optimization (PSO) and cat swarm optimization (CSO), and the new algorithm is referred to as the particle oriented cat swarm optimization model (POCSO). Finally, the performance of the proposed work is compared to other conventional models. In particular, the cost function of POCSO is 6.25%, 6%, 11.7%, 36%, 27%, and 46.42% better than the cost function of whale optimization algorithm (WOA), elephant herd optimization (EHO), moth-flame optimization (MFO), dragonfly algorithm (DA), sealion optimization (SLnO), CSO, and PSO methods, respectively. Also, for IEEE-30 bus system, the best value of the proposed work is 7.46%, 5.41%, 16.30%, 14.88%, 17.60%, 13.86%, 15.21%, 17.49%, and 4.27% better than that of the PSO, CSO, SLnO, DA, MFO, EHO, WOA, multiagent glowworm swarm optimization (MAGSO), and Harris hawks optimization-based feed-forward neural network (HHO-FNN) methods, respectively.

Interpreting the effects of shale rock properties on seismic anisotropy by statistical and machine learning methods

The elastic anisotropy of unconventional shale reservoirs plays a crucial role in their economic hydrocarbon production. The strong anisotropy of organic-rich shale rocks, characterized by clay platelets, elongated pores, and organic material, presents challenges for traditional reservoir characterization methods, which are designed for isotropic or weakly anisotropic conventional sandstone reservoirs. To effectively describe the geomechanical behaviors of organic-rich shales and estimate changes in stress caused by hydraulic fracturing, it is necessary to develop practical methods for predicting anisotropy parameters from conventional rock property logs. This study proposes a statistical approach for predicting independent elastic stiffness coefficients for vertically transverse isotropic (VTI) shale from three shale rock properties: clay volume fraction, kerogen volume fraction, and total porosity. The results of this study show that the stiffness coefficients, C11 and C33, are related to P-wave anisotropy and porosity has a greater impact on P-wave anisotropy than clay and kerogen volume fractions. In contrast, changes in clay volume have a more sensitive effect on C44 and C66, providing S-wave anisotropy information that can help detect lithology changes. Furthermore, kerogen variation has a greater influence on both C12 and C13 than the effects of porosity and clay variations. From these relationships, we develop exponential nonlinear regression (ENLR) models for predicting independent elastic stiffness coefficients from three shale rock properties. These models are validated with the Midland Basin log data, and we improve the accuracy of the C13 estimation model using the deep feedforward neural network (DFNN) method. The results show that the statistical approach can help understand the effect of shale rock components on seismic anisotropy variations and better predict the variations in lithology, fluid, and organic contents in shales from anisotropic seismic data. Furthermore, the methodology can provide robust estimates of the stress changes from hydraulic fracturing, supporting the efficient and economically viable production of shale hydrocarbons.

Machine learning modelling and evaluation of jet fires from natural gas processing, storage, and transport

AbstractJet fires and their repercussions play a significant role in catastrophic incidents that typically have a cascading impact in process industries. Several hydrocarbon experiments from 19 papers were incorporated into the current endeavour to develop simulations of jet flames using machine learning (ML) models. Dimensionless characteristics have been used as output and input variables, including mass flow rates, fuel density, jet flame length, and heat release fluxes. When training three layers of the multi‐layer feedforward neural network (MLFFNN) method, a Bayesian regularization backpropagation approach was adopted and evaluated with the radial based functions (RBF) algorithm. Through an optimization procedure, the first and second hidden layers of the MLFFNN have been optimized to include 10 and five neurons, respectively. The RBF algorithm with 40 neurons in a single layer has been computed using the same method. The best mean square error (MSE) validation results for RBF and MLFFNN were 0.006 and 0.0002, respectively, for 40 and 100 epochs. The MLFFNN and RBF models' respective regression statistical analysis outputs were 0.9949 and 0.9645. The ML method has been identified as a potentially useful technique for precisely predicting the geometrical and radiative characteristics of jet flames.

A new learning rate based on Andrei method for training feed-forward artificial neural networks

In this paper we developed a new method for computing learning rate for Back-propagation algorithm to train a feed-forward neural networks. Our idea is based on the approximating the inverse Hessian matrix for the error function originally suggested by Andrie. Experimental results show that the proposed method considerably improve the convergence rate of the Back-propagation algorithm for the chosen test problem.

Application of a multi-layer feedforward neural network to predict fracture density in shale oil, Junggar Basin, China

Fengcheng Formation in the Mabei Slope of Junggar Basin has low porosity and permeability. However, fractures are well developed, representing an effective storage space for shale oil. Core and experimental data show that the shale oil reservoir of Fengcheng Formation positively correlates with oil content and fractures. And the fracture density has a good quantitatively positive correlation with crude oil production from the production data. Fengcheng Formation has been significantly enriched and accumulated with shale oil due to fractures serving as reservoirs and seepage channels. Therefore, quantitative prediction of fractures is the key to finding high production areas of shale oil in the Fengcheng Formation. The purpose of this study is to extract the seismic attributes that are sensitive to shale oil reservoir fractures. These attributes include curvature, deep learning fracture detection, maximum likelihood, eigenvalue coherence, and variance cube. Furthermore, a seismic multi-attribute fracture density prediction model is trained at the well point using a feedforward neural network method, and the spatial distribution of fracture density is predicted. The results show that the predicted fracture density is consistent with the formation micro imaging logs in the area. Simultaneously, combined with the understanding of the quantitative relationship between fracture density and shale oil production, quantitative prediction results of fracture density could provide the basis for determining the distribution and optimal location of high-quality shale oil wells in the study area. This study will serve as a benchmark for identifying fractures in shale oil reservoirs worldwide.

Prediction of hot metal temperature in a blast furnace iron making process using multivariate data analysis and machine learning methodology

The feed-forward back propagation neural (FFBPN) network method and multivariate data analysis are used to present a new approach for predicting the health of a blast furnace in the form of hot metal temperature (HMT), which is a crucial parameter to control the stable flow of hot metal production while avoiding major danger incidents during the ironmaking process. The health status also appears to predict the performance level of BF at a premature time, allowing the operator to take necessary steps to avoid BF deterioration. The BF’s health status designates the stability or instability of the BF, which may arise during the manufacturing process of hot molten iron, and is used to find the fault. In this paper, the health status of BF was determined with the help of a FFBPN and correlation matrix. This was done with Matlab (Version 2018Rb) software that uses data pre-processing, variable reduction, and a selective attribute of a data set. The FFBPN model has been trained, tested, and validated, and it has got 96% correlation coefficient of HMT prediction of combination of all data sets. The predicted HMT using several actual process data sets has been helpful in identifying the process irregularity in BF.

A Neural Network Model for Estimation of Failure Stresses and Strains in Cohesive Soils

In this article, a set of neural networks for the prediction of the stresses and the corresponding strains at failure of cohesive soils when subjected to a load of a shallow foundation are presented. The data are acquired via Monte Carlo analyses for different types of loadings and stochastic input material variabilities, and by adopting the clayey soil domain and modified Cam Clay material yield function. The mathematical functions for the estimation of the failure stresses and strains are computed with the feed forward neural network method (FNN). It is demonstrated that the accuracy of the derived relations is in the order of a maximum relative error of 10−5 in all monitored output variables. In addition, the number of training epochs required for convergence is relatively low and this means that the computational and data costs for the construction of the FNN are low. The critical input variable for the estimation of the most unfavorable situations is the Karhunen Loeve series expansion for porous analyses, while for non-porous analyses the constant distribution over depth is the one that provides more critical estimations for the monitored output variables of stresses and strains at failure. This set of functions can estimate the aforementioned variables of the footing settlement in clays with high accuracy; consequently, it can be an important tool for geotechnical engineering design, especially in providing the largest stress allowed from the foundation.

Neural-Network-Based Nonlinear Model Predictive Control of Multiscale Crystallization Process

The purpose of this study was to develop an integrated control strategy for multiscale crystallization processes. An image analysis method using a deep learning neural network is used to measure the fine-scale information of the crystallization process, and the mathematical statistical method is adopted to obtain the mean size of the crystal population. A feedforward neural network is subsequently trained and employed in a nonlinear model predictive control formulation to obtain the optimal profile of the manipulated variable. The effectiveness of the proposed nonlinear model predictive control method is evaluated using alum cooling crystallization experiments. Experimental results demonstrate benefits of the proposed combination of feedforward neural network and nonlinear model predictive control method for the multiscale crystallization process.

Deep Learning and Machine Learning with Grid Search to Predict Later Occurrence of Breast Cancer Metastasis Using Clinical Data.

Background: It is important to be able to predict, for each individual patient, the likelihood of later metastatic occurrence, because the prediction can guide treatment plans tailored to a specific patient to prevent metastasis and to help avoid under-treatment or over-treatment. Deep neural network (DNN) learning, commonly referred to as deep learning, has become popular due to its success in image detection and prediction, but questions such as whether deep learning outperforms other machine learning methods when using non-image clinical data remain unanswered. Grid search has been introduced to deep learning hyperparameter tuning for the purpose of improving its prediction performance, but the effect of grid search on other machine learning methods are under-studied. In this research, we take the empirical approach to study the performance of deep learning and other machine learning methods when using non-image clinical data to predict the occurrence of breast cancer metastasis (BCM) 5, 10, or 15 years after the initial treatment. We developed prediction models using the deep feedforward neural network (DFNN) methods, as well as models using nine other machine learning methods, including naïve Bayes (NB), logistic regression (LR), support vector machine (SVM), LASSO, decision tree (DT), k-nearest neighbor (KNN), random forest (RF), AdaBoost (ADB), and XGBoost (XGB). We used grid search to tune hyperparameters for all methods. We then compared our feedforward deep learning models to the models trained using the nine other machine learning methods. Results: Based on the mean test AUC (Area under the ROC Curve) results, DFNN ranks 6th, 4th, and 3rd when predicting 5-year, 10-year, and 15-year BCM, respectively, out of 10 methods. The top performing methods in predicting 5-year BCM are XGB (1st), RF (2nd), and KNN (3rd). For predicting 10-year BCM, the top performers are XGB (1st), RF (2nd), and NB (3rd). Finally, for 15-year BCM, the top performers are SVM (1st), LR and LASSO (tied for 2nd), and DFNN (3rd). The ensemble methods RF and XGB outperform other methods when data are less balanced, while SVM, LR, LASSO, and DFNN outperform other methods when data are more balanced. Our statistical testing results show that at a significance level of 0.05, DFNN overall performs comparably to other machine learning methods when predicting 5-year, 10-year, and 15-year BCM. Conclusions: Our results show that deep learning with grid search overall performs at least as well as other machine learning methods when using non-image clinical data. It is interesting to note that some of the other machine learning methods, such as XGB, RF, and SVM, are very strong competitors of DFNN when incorporating grid search. It is also worth noting that the computation time required to do grid search with DFNN is much more than that required to do grid search with the other nine machine learning methods.

Forecasting the crude oil prices with an EMD-ISBM-FNN model

In this paper, an improved slope-based method (ISBM) based on empirical mode decomposition (EMD) and feed-forward neural network (FNN) method, namely, the EMD-ISBM-FNN method is introduced to decompose and forecast the crude oil prices. Firstly, the ISBM-based EMD method is used to decompose the time series of Brent crude oil prices into several IMFs (intrinsic mode functions) and residuals rn(t). Then IMFs and residuals rn(t) are inputted into the FNN model as input layer neurons, which are trained and integrated by the FNN model to study the relationship between the output values of the FNN and actual values. In order to verify the forecasting results of the EMD-ISBM-FNN model, two research frameworks and three strategies are designed, and the EMD-FNN model and the FNN model as the benchmark models are constructed to compare their forecasting results. The research shows that the EMD-ISBM-FNN model proposed in this paper has the best forecasting effect under the three strategies, and the research framework of this paper is better than the previous scholars' research frameworks, too.

Evaluation of artificial neural network methods to forecast short-term solar power generation: a case study in Eastern Mediterranean Region

Solar power forecasting is substantial for the utilization, planning, and designing of solar power plants. Global solar irradiation (GSI) and meteorological variables have a crucial role in solar power generation. The ever-changing meteorological variables and imprecise measurement of GSI raise difficulties for forecasting photovoltaic (PV) output power. In this context, a major motivation appears for the accurate forecast of GSI to perform effective forecasting of the short-term output power of a PV plant. The presented study comprises of four artificial neural network (ANN) methods; recurrent neural network (RNN) method, feedforward backpropagation neural network (FFBPNN) method, support vector regression (SVR) method, and long short-term memory (LSTM) for daily total GSI prediction of Tarsus by using meteorological data. Moreover, this study proposes a model that utilizes the predicted daily GSI for output power forecasting of a grid-connected PV plant. The obtained results are compared with the output power generation data of a 350 kW solar power plant. The results are evaluated with the performance indices as mean absolute percentage error (MAPE), normalized root mean squared error (NRMSE), weighted mean absolute error (WMAE), and normalized mean absolute error (NMAE). FFBPNN method is chosen with the best results of MAPE 7.066%, NMAE 3.629%, NRMSE 4.673%, and WMAE 5.256%.

An accuracy-maximization learning framework for supervised and semi-supervised imbalanced data

While we attempt to develop the balanced error rate (BER) minimization learning framework for randomized learning of feedforward neural networks to deal with imbalanced datasets, it remains unclear whether the BER minimization learning framework can be effectively extended into its semi-supervised version. This paper proposes a new concept of accuracy maximization for randomized learning methods on imbalanced datasets for the first time, and theoretically proves that it is equivalent to the minimization of the generalized BER for the use of the selected neural networks. In particular, accuracy maximization can be easily extended to semi-supervised scenarios as its semi-supervised version is proved to be linearly dependent on its original. In this paper, based on the proposed accuracy maximization concept, we propose an accuracy-maximization learning framework, and further develop a new accuracy-maximization extreme learning machine AMELM by taking Extreme Learning Machine (ELM) as a typical randomized learning method for feedforward neural networks so as to handle challenging data issues such as the class imbalance and label scarcity. It is worth noting that the proposed accuracy maximization based framework is not only suitable for ELM, but can be extended to different randomized learning methods, such as Random Vector Functional Link Network (RVFL), and Schmidt Neural Network (SNN) for supervised and semi-supervised imbalanced data. The efficacy of AMELM is tested on extensive benchmark datasets. Experimental results show that AMELM can achieve satisfactory performances on labeled or partially labeled imbalanced data. Also, AMELM obtains at least comparable classification performance to other baseline methods yet has fewer hyperparameters to tune, showing its potential for practical applications.