Feed-forward Neural Network Model Research Articles

An Improvised Approach of Deep Learning Neural Networks in NLP Applications

Abstract: In recent years, natural language processing (NLP) has drawn a lot of interest for its ability to computationally represent and analyze human language. Its uses have expanded to include machine translation, email spam detection, information extraction, summarization, medical diagnosis, and question answering, among other areas. The purpose of this research is to investigate how deep learning and neural networks are used to analyze the syntax of natural language. This research first investigates a feed-forward neural network-based classifier for a transfer-based dependent syntax analyzer. This study presents a long-term memory neural network-based dependent syntactic analysis paradigm. This model, which will serve as a feature extractor, is based on the feed-forward neural network model mentioned before. After the feature extractor is learned, we train a recursive neural network classifier that is optimized by sentences using a long short-term memory neural network as a classifier of the transfer action and the characteristics retrieved by the syntactic analyzer as its input. Syntactic analysis replaces the method of modeling independent analysis with one that models the analysis of the entire sentence as a whole. The experimental findings demonstrate that the model has improved its performance more than the benchmark techniques.

Study of Flow Patterns in a Moving Bed Reactor for Chemical Looping Combustion Based on Machine Learning Methods

Abstract A tower-type moving bed can be used as the air reactor in a chemical looping combustion system because of its low-pressure drop and smooth operation. In our previous simulation, a quasi-two-dimensional numerical model was established using discrete element method (DEM) approach to investigate the velocity and solid residence time distributions in the moving bed. In this work, the flow patterns under different operating and structural parameters are studied and optimized via machine learning methods. The random Forest regression model is applied to evaluate the importance of each variable to the solid flow pattern, while the feed forward neural network is applied to buildup a high-accuracy model to predict the solid axial velocity in the moving bed without the requirement to understand the physical mechanisms. Results show that the solid mass flux has the least impact on the mass flow index, while the axial position has the dominant influence and what comes next is the wedge angle, reactor angle, and ratio of down-comer diameter to reactor diameter. Further, based on the established feed forward neural network model, relation between the effective transition position and structural parameters of the moving bed is built, which provides valuable guidance for optimization of the reactor configuration.

FFM: Flood Forecasting Model Using Federated Learning

Floods are one of the most common natural disasters that occur frequently causing massive damage to property, agriculture, economy and life. Flood prediction offers a huge challenge for researchers struggling to predict floods since long time. In this article, flood forecasting model using federated learning technique has been proposed. Federated Learning is the most advanced technique of machine learning (ML) that guarantees data privacy, ensures data availability, promises data security, and handles network latency trials inherent in prediction of floods by prohibiting data to be transferred over the network for model training. Federated Learning technique urges for onsite training of local data models, and focuses on transmission of these local models on the network instead of sending huge data set towards central server for local model aggregation and training of global data model at the central server. In this article, the proposed model integrates locally trained models of eighteen clients, investigates at which station flooding is about to happen and generates flood alert towards a specific client with five days lead time. A local feed forward neural network (FFNN) model is trained at the client station where the flood has been expected. Flood forecasting module of local FFNN model predicts the expected water level by taking multiple regional parameters as input. The dataset of five different rivers and barrages has been collected from 2015 to 2021 considering four aspects including snow melting, rainfall-runoff, flow routing and hydrodynamics. The proposed flood forecasting model has successfully predicted previous floods happened in the selected zone during 2010 to 2015 with 84 % accuracy.

Prediction and Optimization of the Thermal Properties of TiO2/Water Nanofluids in the Framework of a Machine Learning Approach

In this study, comparing multiple models of machine learning, a multiple linear regression (MLP), multilayer feed-forward artificial neural network (BP) model, and a radial-basis feed-forward artificial neural network (RBF-BP) model are selected for the optimization of the thermal properties of TiO₂/water nanofluids. In particular, the least squares support vector machine (LS-SVM) method and radial basis support vector machine (RB-SVM) method are implemented. First, curve fitting is performed by means of multiple linear regression in order to obtain bivariate correlation functions for thermal conductivity and viscosity of the nanofluid. Then the aforementioned models are used for a predictive analysis of the dependence of its thermal conductivity and viscosity on temperature and volume fraction. The results show that the least squares support vector machine (LS-SVM) has a prediction accuracy higher than the other models. The model predicts the thermal conductivity of TiO₂/water MSE = 1.0853 × 10⁻⁶, R²= 0.99864, MAE = 0.00092, RMSE = 0.00104, and the viscosity of TiO₂/water MSE = 8.1397 × 10⁻⁶, R²= 0.99995, MAE = 0.00074, RMSE = 0.0009.

The effect of amplitude modification in S-shaped activation functions on neural network regression

Time series forecasting using multilayer feed-forward neural networks (FNN) is potential to give high accuracy. Several factors influence the accuracy. One of them is the choice of activation functions (AFs). There are several AFs commonly used in FNN with their specific characteristics, such as bounded type AFs. They include sigmoid, softsign, arctan, and tanh. This paper investigates the effect of the amplitude in the bounded AFs on the FNNs’ accuracy. The theoretical investigations use simplified FNN models: linear equation and linear combination. The results show that the higher amplitudes give higher accuracy than typical amplitudes in softsign, arctan, and tanh AFs. However, in sigmoid AF, the amplitude changes do not influence the accuracy. These theoretical results are supported by experiments using the FNN model for time series prediction of 10 foreign exchanges from different continents compared to the US dollar. Based on the experiments, the optimum amplitude of the AFs should be high, that is greater or equal to 100 times of the maximum input values to the FNN, and the accuracy gains up to 3–10 times.

A rapid method of identifying mastitis degrees of bovines based on dielectric spectra of raw milk

AbstractBovine mastitis is the most complex and costly disease in the dairy industry worldwide. Somatic cell count (SCC) is accepted as an international standard for diagnosing mastitis in cows, but most instruments used to detect SCC are expensive, or the detection speed is very low. To develop a rapid method for identifying mastitis degree, the dielectric spectra of 301 raw milk samples at three mastitis grades, i.e., negative, weakly positive, and positive grades based on SCC, were obtained in the frequency range of 20–4500 MHz using coaxial probe technology. Variable importance in the projection method was used to select characteristic variables, and principal component analysis (PCA) and partial least squares (PLS) were used to reduce data dimension. Linear discriminant analysis, support vector classification (SVC), and feed-forward neural network models were established to predict the mastitis degrees of cows based on 22 principal components and 24 latent variables obtained by PCA and PLS, respectively. The results showed that the SVC model with PCA had the best classification performance with an accuracy rate of 95.8% for the prediction set. The research indicates that dielectric spectroscopy technology has great potential in developing a rapid detector to diagnose mastitis in cows in situ or online.

TRANSFORMING CONSUMER BEHAVIOR TO NEW PARADIGMS THROUGH DEEP LEARNING APPLICATIONS

Direct-to-consumer (DTC) businesses are gaining popularity as a way to reach a larger number of customers and better suit their needs. Vertical brands are distinguished by their metamorphosis, in which they offer their products straight from the manufacturer to consumers without the use of distribution intermediaries, like in traditional business models. They're obliterating themselves on virtual platforms and undermining their old linear sales processes in the process. In the current scenario, the ability of connectionist models to explain consumer behavior, with a focus on the feed-forward neural network model, should be emphasized, and the possibility of expanding the implications of ANN (Artificial Neural Network) for predicting buying behavior for DTC (direct-to-consumer) brands should be explored. To forecast consumer loyalty as a critical feature of consumer behavior, a variety of neural network models of various complexity are constructed. When compared to the more standard logistic regression approach, neural networks outperform logistic regression in predicting customer loyalty. Utilitarian and Informational Reinforcement factors, both independently determined, are shown to contribute significantly to the explanation of consumer choice. The potential of connectionist models for predicting and explaining consumer behavior is discussed, and future research directions are proposed for investigating the predictive and explanatory capacity of connectionist models, such as neural network models, and their integration into consumer behavior analysis within the theoretical framework.

TRANSFORMING CONSUMER BEHAVIOR TO NEW PARADIGMS THROUGH DEEP LEARNING APPLICATIONS

Direct-to-consumer (DTC) businesses are gaining popularity as a way to reach a larger number of customers and better suit their needs. Vertical brands are distinguished by their metamorphosis, in which they offer their products straight from the manufacturer to consumers without the use of distribution intermediaries, like in traditional business models. They're obliterating themselves on virtual platforms and undermining their old linear sales processes in the process. In the current scenario, the ability of connectionist models to explain consumer behavior, with a focus on the feed-forward neural network model, should be emphasized, and the possibility of expanding the implications of ANN (Artificial Neural Network) for predicting buying behavior for DTC (direct-to-consumer) brands should be explored. To forecast consumer loyalty as a critical feature of consumer behavior, a variety of neural network models of various complexity are constructed. When compared to the more standard logistic regression approach, neural networks outperform logistic regression in predicting customer loyalty. Utilitarian and Informational Reinforcement factors, both independently determined, are shown to contribute significantly to the explanation of consumer choice. The potential of connectionist models for predicting and explaining consumer behavior is discussed, and future research directions are proposed for investigating the predictive and explanatory capacity of connectionist models, such as neural network models, and their integration into consumer behavior analysis within the theoretical framework.

Categorical Embeddings for Tabular Data using PyTorch

Deep learning has received much attention for computer vision and natural language processing, but less for tabular data, which is the most prevalent type of data used in industry. Embeddings offer a solution by representing categorical variables as continuous vectors in lowdimensional space. PyTorch provides excellent support for GPU acceleration and pre-built functions and modules, making it easier to work with embeddings and categorical variables. In this research paper, we apply a feedforward neural network model in PyTorch to a multiclass classification problem using the Shelter Animal Outcome dataset. We calculate the probability of an animal's outcome belonging to each of the 5 categories. Additionally, we explore feature importance using two common techniques: MDI and permutation. Understanding feature importance is crucial for building better models, improving performance, and interpreting and communicating results. Our findings demonstrate the usefulness of embeddings and PyTorch for deep learning with tabular data and highlight the importance of feature selection for building effective machine learning models.

Personal Identification Using Palm Features Recognition

Personal recognition is meant for finding a way for establishment of connections between the person and his/her biometrical features. Such system is depending various data types such as facial images, voice and limbs. In this paper, palm print recognition is made using deep learning paradigms such as feed forward neural network (FFNN). The palm features are extracted by tracking the principal lines of palm skin. This involves performing of pixel to pixel analysis by comparing pixel value with its four sides neighbors. FFNN model is tuned up using ABC-KNN algorithm then used for classification. The proposed system has yielded good recognition accuracy score of 98.66%.

A deep learning based feed forward artificial neural network to predict the K-barriers for intrusion detection using a wireless sensor network

Wireless Sensor Networks (WSN) has a wide range of opportunities and can be applied to almost every part of our life. Intrusion detection and monitoring across disputed areas and on army bases are two of the most important uses of WSN. An innovative method that precisely calculates many barriers involved in the process for quick identification and prevention of intrusions. The relevance and risk level of the chosen characteristics were assessed in this paper using feature importance and feature sensitivity metrics. This paper develops a deep learning-based feed-forward artificial neural network model that enables precise estimates of the k-barrier count for effective intrusion detection and mitigation. The area of the region of interest, sensor sensing area, sensing transmission area, and many sensors are the four potential characteristics that were used to learn and assess the feed-forward ANN model. The various attributes are extracted via Monte Carlo simulation. This research found that the method correctly forecasts the number of barriers with Root Mean Square Error and correlation coefficient. The proposed methodology is more efficient than the other standard methods.

A new method for building single feedforward neural network models for multivariate static regression problems: a combined weight initialization and constructive algorithm

A new method for building single feedforward neural network models for multivariate static regression problems: a combined weight initialization and constructive algorithm

Gradient-Harmonizing-Based Deep Learning for Thermophysical Properties of Carbon Dioxide

Carbon dioxide presents many unique advantages for cooling and power cycles under supercritical or near-critical conditions, where the characterization of thermophysical properties is a daunting task. The present study proposes different deep feedforward neural network (DFNN) models for property evaluations of carbon dioxide. The all-in-one DFNN model appears acceptable for enthalpy, entropy, and thermal conductivity, but it exhibits poor performance in density, speed of sound, viscosity, and constant-pressure specific heat. The specific DFNN model presents limited improvement in the near-critical and pseudoboiling regions, where steep property gradients occur. To alleviate the situation, the sampling data are divided into easy samples and hard samples. Easy samples are data that have small-gradient norm and can be well fitted, whereas hard samples are those with large-gradient norms and are difficult to fit. The gradient-harmonizing method is proposed to solve the imbalance between hard and easy samples by rectifying their gradient contribution and assigning different weights. The resultant models show significantly improved performance as compared to the existing methods in the literature, with less than 0.4% in absolute average relative errors for the studied properties. The gradient-harmonizing algorithm can potentially be adopted to property evaluations of other substances involved in aerospace propulsion.

A Comparison Of Feed Forward Neural Network Models And Time Series Models For Forecasting Turkey's Monthly Dairy Exports To Iraq

A Comparison Of Feed Forward Neural Network Models And Time Series Models For Forecasting Turkey's Monthly Dairy Exports To Iraq

Optimization of homogeneous charge compression ignition combustion in a light-duty diesel engine operated using ethyl acetate-gasoline blends

The low temperature combustion mode of homogeneous charge compression ignition eliminates particulate matter and oxides of nitrogen (NO x) emissions trade-off that prevails in high-temperature, diffusion-controlled conventional diesel combustion (CDC). In the present research, the significant challenge of narrow operating load range that hinders the commercial implementation of light-duty HCCI engines was overcome by employing ethyl acetate-gasoline blends. The gasoline concentration in test fuels was reduced in 10% decrements, from 84% to 24%, to replace it with ethyl acetate. The use of ethyl acetate, a renewable fuel, can help solve the energy crisis due to the rapid depletion of fossil fuels. An ignition improver was blended in the test fuels in a predetermined amount of 6% so that combustion stability was not hampered at lower loads. Parametric investigations were conducted to study the effect of progressively increasing ethyl acetate in test fuels on HCCI combustion, performance, and emissions. The machine learning tool of artificial neural network was implemented to learn the behavior of the test engine, considering load and fuel composition as input variables. The feedforward artificial neural network models were developed to predict the start of combustion, combustion phasing, indicated thermal efficiency, and emissions of carbon monoxide, soot, NO x, and unburned hydrocarbon (HC). A multi-objective optimization was performed to arrive at the best operating condition by integrating artificial neural network models with the genetic algorithm. All the developed artificial neural network models could predict responses with acceptable accuracy. The genetic algorithm indicated that the optimum point of the operation was at 80% load and 65% ethyl acetate in the test fuels. Experiments were conducted to validate the optimal HCCI conditions that resulted in 27% higher indicated thermal efficiency, 54% lower HC+NOx, and 99% lower soot emissions than CDC. Overall, the present study demonstrated the benefits of considering ethyl acetate as a fuel to improve HCCI engine metrics of off-road diesel engines.

Prediction of ultimate conditions and stress–strain behaviour of steel-confined ultra-high-performance concrete using sequential deep feed-forward neural network modelling strategy

Recognising that ultra-high-performance concrete (UHPC) is gaining momentum in structural applications, providing an accurate confinement model is essential to developing a reliable design of UHPC structural members. However, very limited number of models are currently available and these models were empirically formulated and calibrated upon limited test data obtained by the originators of the models. The significant cost associated with comprehensive experimental testing motivates the exploration of cheaper and more efficient data-driven based machine learning approach. This study proposes a sequential artificial neural network (ANN) framework to develop such a data-driven confinement model, incorporating a comprehensive database of 228 axially loaded UHPC columns compiled from available literature. Three deep feed-forward neural network models were established to predict the ultimate stress, ultimate strain and stress–strain behaviour of confined UHPC. The results show that the proposed ANN-based ultimate condition models provide a more robust prediction results compared to the existing design-oriented models for confined UHPC. The stress–strain behaviour, predicted using the proposed ANN model, shows high accuracy levels in capturing different types of stress–strain curves as well as reasonably matching results with those experimentally measured responses. The encouraging outcomes in this study suggest that the proposed models are capable of providing rapid prediction tools that will help to facilitate the on-demand design of UHPC structural components and systems.

Application of the PSO–neural network in rainfall–runoff modeling

Abstract In this study, rainfall–runoff (R–R) models were developed by assembling Particle Swarm Optimization (PSO) with the Feed Forward Neural Network (FFNN) and the Wavelet Neural Network (WNN). Performances of the model were compared with the wavelet ensembled neural network (WNN) and the conventional FFNN. The data from 1981 to 2005 were used for calibration and from 2006 to 2014 for validation of the models. Different combinations of rainfall and runoff were considered as inputs to the PSO–FFNN model. The fitness value and computational time of all the combinations were computed. Input combination was selected based on the lowest fitness value and lowest computational time. Four R–R models (FFNN, WNN, PSO–FFNN and PSO–WNN) were developed with the best input combination. The performance of the models was evaluated using statistical parameters (Nash–Sutcliffe Efficiency (NSE), D and root mean square error (RMSE)) and parameters vary in the range of (0.86–0.90), (0.95–0.97) and (68.87–84.37), respectively. After comparing the performance parameters and computational time of all four models, it was found that the PSO–FFNN model gave better values of NSE (0.89), D (0.97), RMSE (68.87) and less computational time (125.42 s) than other models. Thus, the PSO–FFNN model was better than the other three models (FFNN, WNN and PSO–WNN).

Congestion Intrusion Detection-Based Method for Controller Area Network Bus: A Case for KIA SOUL Vehicle

In the vehicle industry, connectivity and autonomy are becoming increasingly important features. One of the most used protocols for in-vehicle communication is the Controller Area Network (CAN) bus which manages the communication between networked components. However, the CAN bus, despite its critical importance, lacks sufficient security features to protect its network as well as the overall car system. Thus, vehicle network security is becoming increasingly crucial. Methods of intrusion detection help to improve the security of the in-vehicle network. This work aims to provide a model that enables effective detection of attacks such as fuzzy, DoS, and impersonation using the Deep Feedforward Neural Network (DeepFNN) model as well as the Long Short-Term Memory model. Moreover, the LSTM model presents the most satisfying outcome in terms of precision and recall metrics.

Boosting tissue-specific prediction of active cis-regulatory regions through deep learning and Bayesian optimization techniques

BackgroundCis-regulatory regions (CRRs) are non-coding regions of the DNA that fine control the spatio-temporal pattern of transcription; they are involved in a wide range of pivotal processes such as the development of specific cell-lines/tissues and the dynamic cell response to physiological stimuli. Recent studies showed that genetic variants occurring in CRRs are strongly correlated with pathogenicity or deleteriousness. Considering the central role of CRRs in the regulation of physiological and pathological conditions, the correct identification of CRRs and of their tissue-specific activity status through Machine Learning methods plays a major role in dissecting the impact of genetic variants on human diseases. Unfortunately, the problem is still open, though some promising results have been already reported by (deep) machine-learning based methods that predict active promoters and enhancers in specific tissues or cell lines by encoding epigenetic or spectral features directly extracted from DNA sequences.ResultsWe present the experiments we performed to compare two Deep Neural Networks, a Feed-Forward Neural Network model working on epigenomic features, and a Convolutional Neural Network model working only on genomic sequence, targeted to the identification of enhancer- and promoter-activity in specific cell lines. While performing experiments to understand how the experimental setup influences the prediction performance of the methods, we particularly focused on (1) automatic model selection performed by Bayesian optimization and (2) exploring different data rebalancing setups for reducing negative unbalancing effects.ConclusionsResults show that (1) automatic model selection by Bayesian optimization improves the quality of the learner; (2) data rebalancing considerably impacts the prediction performance of the models; test set rebalancing may provide over-optimistic results, and should therefore be cautiously applied; (3) despite working on sequence data, convolutional models obtain performance close to those of feed forward models working on epigenomic information, which suggests that also sequence data carries informative content for CRR-activity prediction. We therefore suggest combining both models/data types in future works.

Physics-data combined machine learning for parametric reduced-order modelling of nonlinear dynamical systems in small-data regimes

Repeatedly solving nonlinear partial differential equations with varying parameters is often an essential requirement to characterise the parametric dependences of dynamical systems. Reduced-order modelling (ROM) provides an economical way to construct low-dimensional parametric surrogates for rapid predictions of high-dimensional physical fields. This paper presents a physics-data combined machine learning (PDCML) method for non-intrusive parametric ROM in small-data regimes. Proper orthogonal decomposition (POD) is adopted for dimension reduction by deriving basis functions from a limited number of high-fidelity snapshots, and parametric ROM is thus transformed into establishing reliable mappings between the system parameters and the POD coefficients. To overcome labelled data scarcity, a physics-data combined ROM framework is developed to jointly integrate the physical principle and the small labelled data into feedforward neural networks (FNN) via a step-by-step training scheme. Specifically, a preliminary FNN model is firstly fitted via data-driven training, and then the governing physical rules are embedded into the loss function to improve the model interpolation and extrapolation performances through physics-guided training constrained by the labelled data. During the constrained optimisation procedure, dynamic weighting factors are used to adjust the physics-data proportion of the loss functions, aiming at continuously highlighting the physics loss as the primary optimisation objective and keeping the data loss as the constraint. This new PDCML method is tested on a series of nonlinear problems with different numbers of physical variables, and it is also compared with the data-driven ROM, the physics-guided ROM and the traditional projection-based ROM methods. The results demonstrate that the proposed method provides a cost-effective way for non-intrusive parametric ROM via machine learning, and it possesses good characteristics of high prediction accuracy, strong generalisation capability and small data requirement.