Feedforward Network Research Articles

Artificial Neural Networks for Classification Tasks: A Systematic Literature Review

Artificial neural networks (ANNs) have become indispensable tools for solving classification tasks across various domains. This systematic literature review explores the landscape of ANN utilization in classification, addressing three key research questions: the types of architectures employed, their accuracy, and the data utilized. The review encompasses 30 studies published between 2019 and 2024, revealing Convolutional Neural Networks (CNNs) as the predominant architecture in image-related tasks, followed by Multilayer Perceptron (MLP) architectures for general classification tasks. Feed Forward Neural Networks (FFNN) exhibited the highest average accuracy with a 97.12%, with specific studies achieving exceptional results across diverse classification tasks. Moreover, the review identifies digitized images as a commonly utilized data source, reflecting the broad applicability of ANNs in tasks such as medical diagnosis and remote sensing. The findings underscore the importance of machine learning approaches, highlight the robustness of ANNs in achieving high accuracy, and suggest avenues for future research to enhance interpretability, efficiency, and generalization capabilities, as well as address challenges related to data quality.

Comparison and integration of physical and interpretable AI-driven models for rainfall-runoff simulation

Precise streamflow forecasting in river systems is crucial for water resources management and flood risk assessment. The Tagus Headwaters River Basin (THRB) in Spain is a key hydrological hub, providing regulated flow for agricultural, urban, and energy sectors, and facilitating water transfer to the Segura River Basin, a key semi-arid region. Given the basin's near-total allocation of water resources, accurate streamflow simulations are essential to optimize the socio-economic distribution and ensure sustainable management across both interconnected basins. This study conducts a comprehensive evaluation of the Soil and Water Assessment Tool (SWAT+), support vector regression (SVR), feed forward neural network (FFNN), and long short-term memory (LSTM) models in simulating the rainfall-runoff process at four gauging stations within the THRB. An ensemble machine learning technique is then applied to assess improvements in streamflow estimation. Results revealed that AI-based models significantly surpassed the SWAT+ model in performance. Furthermore, the application of an ensemble technique enhanced the precision of rainfall-runoff modeling by 18 to 26% during the calibration period and 4.1 to 9.2% during the validation period, compared to individual AI-based models. Additionally, the SWAT+ model's precision improved by 44 to 74% and 40 to 55% for the respective periods. The use of Shapley Additive Explanations (SHAP) methodology allowed the results of the ensemble with machine learning to be more interpretable by explaining how each model contributes to the prediction. This research offers significant contributions to hydrological modeling, highlighting the importance of ensemble techniques in elevating predictive accuracy for various river basins.

Ứng dụng phương pháp học máy dự báo mức độ căng thẳng tài chính của doanh nghiệp niêm yết tại Việt Nam

This study utilizes the traditional statistical method and machine learning methods to forecast the level of financial distress of listed companies in Vietnam. The data used in the study was collected from 646 listed companies on the HOSE and HNX exchanges during the period from 2012 to 2022. The study experiments with the dataset on 6 machine learning models: Multiple Regression Model (MRL), LASSO, Generalized Additive Model (GAM), Random Forests (RF), Gradient Boosting Regression Trees (GBRT), and a single layer, feed forward neural network (NN), as well as the traditional statistical method. The research results demonstrate a congruence between the outcomes of the traditional method and modern machine learning models. Among these, the model with the highest accuracy is Random Forest (RF) with an accuracy rate of 98.8%. The variables that most influence the financial distress status of companies are TANG, ROA, LTD, NPM. Based on these results, our study proposes recommendations to support informed and effective financial decisions for stakeholders (companies, regulatory agencies, shareholders, creditors, investors).

Hybrid Approach of Multiclassification for Lung Cancer Types Identification

As a widespread health concern, lung cancer requires sophisticated detection techniques for a precise classification. This research proposes an integrated framework for improving how well lung cancer is detected by combining processing methods for images and machine learning. The dataset, which includes normal cases, large-cell carcinoma, squamous cell carcinoma, and adenocarcinoma, allows for a detailed investigation of various forms of lung cancer. K-means clustering is the initial stage of the approach of segmenting complex images. Utilizing the VGG16 model, features are extracted that are important for classification. Several classification models, such as SVM, Logistic Regression, and Feedforward Neural Network, are trained using the combined features from clustering and VGG16. This research goes beyond the dichotomy of benign and malignant cases, investigating in more detail the subtypes of large-cell carcinoma, squamous cell carcinoma, and adenocarcinoma. Including different classes of lung cancer increases the granularity of the classification process, improving diagnostic accuracy and clinical insights. The results highlight how well the suggested strategy works, which is a major advancement in the accurate identification of distinct kinds of lung cancer.

DeepMarkerNet: Leveraging supervision from the Duchenne Marker for spontaneous smile recognition

Distinguishing between spontaneous and posed smiles from videos poses a significant challenge in pattern classification literature. Researchers have developed feature-based and deep learning-based solutions for this problem. To this end, deep learning outperforms feature-based methods. However, certain aspects of feature-based methods could improve deep learning methods. For example, previous research has shown that Duchenne Marker (or D-Marker) features from the face play a vital role in spontaneous smiles, which can be useful to improve deep learning performances. In this study, we propose a deep learning solution that leverages D-Marker features to improve performance further. Our multi-task learning framework, named DeepMarkerNet, integrates a transformer network with the utilization of facial D-Markers for accurate smile classification. Unlike past methods, our approach simultaneously predicts the class of the smile and associated facial D-Markers using two different feed-forward neural networks, thus creating a symbiotic relationship that enriches the learning process. The novelty of our approach lies in incorporating supervisory signals from the pre-calculated D-Markers (instead of as input in previous works), harmonizing the loss functions through a weighted average. In this way, our training utilizes the benefits of D-Markers, but the inference does not require computing the D-Marker. We validate our model’s effectiveness on four well-known smile datasets: UvA-NEMO, BBC, MMI facial expression, and SPOS datasets, and achieve state-of-the-art results.

Research on turbulence model parameters for the prediction of waterjet propulsion pump hydrodynamic performance

Abstract The hydrodynamic performance of waterjet propulsion pumps has an important impact on improving the speed of waterjet propulsion ships. High-precision prediction of the hydrodynamic performance of water-jet propulsion pumps based on CFD is of great significance to improving the pump design quality. Based on the Reynolds-averaged Naiver-Stokes equation, this paper conducts a grid-independence verification on a water jet propulsion pump based on hydrodynamic test data, uses the Latin hypercube sampling method to generate SST k-ω turbulence model parameter space calculation samples, performs steady-state calculation on all samples, analyzes the sensitivity of changes in the parameters of the turbulence model to the prediction of the external characteristics of the waterjet propulsion pump, and finally obtains the influence of changes in the parameters of the turbulence model on the prediction of the external characteristics of the pump. Among them, the sensitivities of the three parameters β *, β1 and a1 are higher. The results show that the prediction accuracy of the external characteristics of the water jet propulsion pump can be effectively improved by correcting these three highly sensitive parameters, and the correction values of the three key parameters are obtained. The prediction error of the external characteristics of the water jet propulsion pump is reduced from 4.3% to less than 2.1%. A feedforward neural network is used to establish a mapping model between the three turbulence model parameters and the head coefficient and power coefficient of the waterjet propulsion pump in order to achieve rapid prediction. The genetic algorithm is applied to optimize the weights, thresholds and other coefficients in the neural network, thereby improving the prediction accuracy. Based on the optimized neural network, the genetic algorithm is used to optimize the three turbulence model parameters, and the optimized coefficients are obtained. The optimized turbulence model parameters are put into the hydrodynamic model for RANS simulation to verify the degree of optimization of the model coefficients by the genetic algorithm.

Identification and Quantification of Common Adulterants in Extra Virgin Olive Oil Using Microwave Dielectric Spectroscopy Aided by Feedforward Neural Networks

Identification and Quantification of Common Adulterants in Extra Virgin Olive Oil Using Microwave Dielectric Spectroscopy Aided by Feedforward Neural Networks

Industrial data-driven machine learning soft sensing for optimal operation of etching tools

Smart Manufacturing, or Industry 4.0, has gained significant attention in recent decades with the integration of Internet of Things (IoT) and Information Technologies (IT). As modern production methods continue to increase in complexity, there is a greater need to consider what variables can be physically measured. This advancement necessitates the use of physical sensors to comprehensively and directly gather measurable data on industrial processes; specifically, these sensors gather data that can be recontextualized into new process information. For example, artificial intelligence (AI) machine learning-based soft sensors can increase operational productivity and machine tool performance while still ensuring that critical product specifications are met. One industry that has a high volume of labor-intensive, time-consuming, and expensive processes is the semiconductor industry. AI machine learning methods can meet these challenges by taking in operational data and extracting process-specific information needed to meet the high product specifications of the industry. However, a key challenge is the availability of high quality data that covers the full operating range, including the day-to-day variance. This paper examines the applicability of soft sensing methods to the operational data of five industrial etching machines. Data is collected from readily accessible and cost-effective physical sensors installed on the tools that manage and control the operating conditions of the tool. The operational data are then used in an intelligent data aggregation approach that increases the scope and robustness for soft sensors in general by creating larger training datasets comprised of high value data with greater operational ranges and process variation. The generalized soft sensor can then be fine-tuned and validated for a particular machine. In this paper, we test the effects of data aggregation for high performing Feedforward Neural Network (FNN) models that are constructed in two ways: first as a classifier to estimate product PASS/FAIL outcomes and second as a regressor to quantitatively estimate oxide thickness. For PASS/FAIL classification, a data aggregation method is developed to enhance model predictive performance with larger training datasets. A statistical analysis method involving point-biserial correlation and the Mean Absolute Error (MAE) difference score is introduced to select the optimal candidate datasets for aggregation, further improving the effectiveness of data aggregation. For large datasets with high quality data that enable model training for more complex tasks, regression models that predict the oxide thickness of the product are also developed. Two types of models with different loss functions are tested to compare the effects of the Mean Squared Error (MSE) and Mean Absolute Percentage Error (MAPE) loss functions on model performance. Both the classification and regression models can be applied in industrial settings as they provide additional information regarding the process outcome. Individually, these models can reduce the number of metrology steps in semiconductor factories, and when developed further, can empower the development of advanced process control strategies.

Remote sensing image classification based on non-linear enhanced attention mechanism

Abstract In recent years, Transformer technology has gradually shown promising applications in the field of computer vision, becoming a research hotspot. However, traditional vision Transformers suffer from significant computational burdens. Although previous studies have attempted to alleviate this issue by reducing the computational load of attention mechanisms, their methods still require improvement. Additionally, past research often overlooked the non-linear representation of values within attention mechanisms, posing another challenge to address. Therefore, this paper proposes a novel attention mechanism aimed at reducing the computational burden of traditional attention mechanisms while enhancing their ability to express non-linearities in values. Furthermore, conventional feedforward neural networks typically output results directly after passing through an activation function, which may limit the model’s representational capacity due to the simplistic structure leading to singular feature information. To address this issue, this study introduces a more robust feature fusion convolutional feedforward network. Through these methods, the aim is to enhance the accuracy of Transformers in the field of remote-sensing image classification.

Enhancing the analysis of murine neonatal ultrasonic vocalizations: Development, evaluation, and application of different mathematical models.

Rodents employ a broad spectrum of ultrasonic vocalizations (USVs) for social communication. As these vocalizations offer valuable insights into affective states, social interactions, and developmental stages of animals, various deep learning approaches have aimed at automating both the quantitative (detection) and qualitative (classification) analysis of USVs. So far, no notable efforts have been made to determine the most suitable architecture. We present the first systematic evaluation of different types of neural networks for USV classification. We assessed various feedforward networks, including a custom-built, fully-connected network, a custom-built convolutional neural network, several residual neural networks, an EfficientNet, and a Vision Transformer. Our analysis concluded that convolutional networks with residual connections specifically adapted to USV data, are the most suitable architecture for analyzing USVs. Paired with a refined, entropy-based detection algorithm (achieving recall of 94.9 % and precision of 99.3 %), the best architecture (achieving 86.79 % accuracy) was integrated into a fully automated pipeline capable of analyzing extensive USV datasets with high reliability. In ongoing projects, our pipeline has proven to be a valuable tool in studying neonatal USVs. By comparing these distinct deep learning architectures side by side, we have established a solid foundation for future research.

DT-Block: Adaptive vertical federated reinforcement learning scheme for secure and efficient communication in 6G

The necessities of security and data sharing have focused on federated learning because of using decentralized data sources. The existing works used federated learning for security, however, it still faces many challenges such as poor security and privacy, computational complexity, etc. In this research, we propose adaptive vertical federated learning using a reinforcement learning approach and blockchain. The proposed work includes three phases: user registration and authentication, machine learning-based client selection, and adaptive secure federated learning. Initially, all the users register their credentials to the cognitive agent, which generates a private key, public key, and random number using a Chaotic Isogenic Post Quantum Cryptography (CIPQC) algorithm. Second, optimal clients are selected for participating in federated learning which improves learning rate and reduces complexity. Here, optimal clients are selected by the Enhanced Multilayer Feed Forward Neural Network (EMFFN) algorithm by considering CSI, RSSI, bandwidth, energy, communication efficiency, and statistical efficiency. Finally, adaptive secure federated learning is performed by the Distributed Distributional Deep Deterministic Policy Gradient (D4PG) algorithm, where the local models are adaptively used by the private strategy based on its sensitivity. The aggregated global models are stored in DT-block (dendrimer tree-based blockchain) which stores the data in a dendrimer tree structure for increasing scalability and reducing search time during data retrieval. The simulation of this research is conducted by NS-3.26 network simulator and the performance of the proposed DT-Block model is estimated based on various performance metrics such as accuracy, delay, loss, f1-score, and security strength this demonstrated that the suggested effort produced better results both in terms of quantitative and qualitative aspects.

Analytical Bounds on the Local Lipschitz Constants of ReLU Networks.

In this article, we determine analytical upper bounds on the local Lipschitz constants of feedforward neural networks with rectified linear unit (ReLU) activation functions. We do so by deriving Lipschitz constants and bounds for ReLU, affine-ReLU, and max pooling functions, and combining the results to determine a network-wide bound. Our method uses several insights to obtain tight bounds, such as keeping track of the zero elements of each layer, and analyzing the composition of affine and ReLU functions. Furthermore, we employ a careful computational approach which allows us to apply our method to large networks such as AlexNet and VGG-16. We present several examples using different networks, which show how our local Lipschitz bounds are tighter than the global Lipschitz bounds. We also show how our method can be applied to provide adversarial bounds for classification networks. These results show that our method produces the largest known bounds on minimum adversarial perturbations for large networks such as AlexNet and VGG-16.

A Human Resource Evaluation and Recommendation System based on Big Data Mining

This investigation presents a paradigm-shifting Human Resource (HR) Assessment and Recommendation Framework, leveraging progressed huge information mining procedures. Drawing bits of knowledge from cybersecurity, healthcare, education, and further detecting spaces, our framework utilizes a different cluster of calculations, counting Arbitrary Forests, Support Vector Machines, K-Means Clustering, and a Feedforward Neural Organize. The comparative investigation uncovers the Feedforward Neural Network as the standout entertainer, emphasizing its various levels including learning for complicated design acknowledgement inside HR measurements. Uniquely, this framework draws on methodologies and insights from varied domains such as cybersecurity, healthcare, and education, applying these rich, interdisciplinary perspectives to HR analytics. This cross-pollination of ideas enables the framework to adopt sophisticated data mining and pattern recognition techniques that are not traditionally utilized within HR, offering new avenues for detecting and interpreting complex employee data patterns. Result values illustrate the system’s adequacy, with a precision of 88%, an accuracy of 90%, a review of 87%, and an F1 score of 88%. These measurements emphasize the system’s capacity to comprehensively assess worker execution, giving exact suggestions for key HR decision-making. Ethical contemplations, innovation acknowledgement, and custom-fitted proposal frameworks, propelled by related works, are coordinates to guarantee the system’s reasonability over assorted organizational settings. This research contributes to the advancing scene of HR administration, offering a spearheading arrangement for organizations looking for data-driven, comprehensive, and morally sound approaches to workforce optimization.

Probabilistic Estimation of Parameters for Lubrication Application with Neural Networks

This paper investigates the use of neural networks to predict characteristic parameters of the grease application process pressure curve. A combination of two feed-forward neural networks was used to estimate both the value and the standard deviation of selected features. Several neuron configurations were tested and evaluated in their capability to make a probabilistic estimation of the lubricant’s parameters. The value network was trained with a dataset containing the full set of features and with a dataset containing its average values. As expected, the full network was able to predict noisy features well, while the average network made smoother predictions. This is also represented by the networks’ R2 values which are 0.781 for the full network and 0.737 for the mean network. Several further neuron configurations were tested to find the smallest possible configuration. The analysis showed that three or more neurons deliver the best fit over all features, while one or two neurons are not sufficient for prediction. The results showed that the grease application process pressure curve via pressure valves can be estimated by using neural networks.

Assessment of sealing systems impact on the vibration and environmental safety of rotary machines

Energy saving control algorithms of centrifugal fans/pumps are based on the use of the frequency-controlled induction motor drives and pressure or flow rate sensors, the costs of which are comparable to the cost of the fans/pumps for low-power applications. The paper develops a new and simple estimation approach of the pressure and flow rate, utilising the measured Root Mean Square (RMS) value of the stator current, estimated motor’s input active power, reference stator voltage frequency and feed-forward backpropagation artificial neural network. The error percentage for both flow rate and pressure in experimental and estimated data is within the range of ±5%, which conforms to the ISO 13348 standard. A test rig for the rapid control prototyping of the fan is designed, and necessary design and test procedures are developed. The estimation approach is verified experimentally and demonstrates better estimation accuracy compared to the existing and possible similar simple approaches. The developed algorithm can be easily embedded into the industrial variable frequency drives without any hardware changes.

Improved Flow Rate and Pressure ANN Estimators for a Centrifugal Fan with Induction Motor Drive

Energy saving control algorithms of centrifugal fans/pumps are based on the use of the frequency-controlled induction motor drives and pressure or flow rate sensors, the costs of which are comparable to the cost of the fans/pumps for low-power applications. The paper develops a new and simple estimation approach of the pressure and flow rate, utilising the measured Root Mean Square (RMS) value of the stator current, estimated motor’s input active power, reference stator voltage frequency and feed-forward backpropagation artificial neural network. The error percentage for both flow rate and pressure in experimental and estimated data is within the range of ±5%, which conforms to the ISO 13348 standard. A test rig for the rapid control prototyping of the fan is designed, and necessary design and test procedures are developed. The estimation approach is verified experimentally and demonstrates better estimation accuracy compared to the existing and possible similar simple approaches. The developed algorithm can be easily embedded into the industrial variable frequency drives without any hardware changes.

Neural networks for ID gap orbit distortion compensation in PETRA III

Undulators are used in storage rings to produce extremely brilliant synchrotron radiation. In the ideal case, a perfectly tuned undulator always has a first and second field integrals equal to zero. But, in practice, field integral changes during gap movements can never be avoided for real-life devices. As they significantly impact the circulating electron beam, there is the need to routinely compensate such effects. Deep Neural Networks can be used to predict the distortion in the closed orbit induced by the undulator gap variations on the circulating electron beam. In this contribution several state-of-the-art deep learning algorithms were trained on measurements from PETRA III. The different architecture performances are then compared to identify the best model for the gap-induced distortion compensation. It is found that realistic data, with several gaps moving simultaneously to arbitrary gaps must be used to train the neural networks. The deep feed-forward neural network was found to be more effective than the recurrent and convolutional neural networks.

Email Spam Detection Using a Hybrid Approach of Feedforward Neural Network and Penguin Optimization Algorithm

The electronic communication of today is beset by unwanted junk messages. These are not just a nuisance; they waste our time and energy and clog up our inboxes, which are supposed to be, in many ways, our digital front doors. But how do we keep them from coming? Well, as today's article makes clear, the situation is hardly straightforward. There are various methods that can be used, both traditional and modern, and each has its own set of pros and cons. One method, however, stands out in today's article as one that we think might work well in practice, the good old-fashioned neural network. The method's novelty derives from its use of neural networks, a type of machine learning, in conjunction with the POA. The POA is a powerful optimization algorithm that can perform very well when fine-tuning the weights and biases of a neural network. We therefore used the POA to optimize these two crucial components of our model and achieved a surprising level of accuracy—98.7%. This figure surpasses the next nearest competitor by 2% and demonstrates the efficacy of our method in spam detection while also highlighting the POA's potential in this field. The overall proposed method pushes the spam-detection field forward, but it also demonstrates something even broader: the power of hybrid approaches in machine learning. Neural networks are great—but you have to train them properly, and the use of a particle swarm for that purpose is a neat idea. But in the end, a hybrid spam filter is still a spam filter, and this one does what it's supposed to do better than many rivaling methods.

Spiking Neural Networks for event-based action recognition: A new task to understand their advantage

Spiking Neural Networks (SNN) are characterised by their unique temporal dynamics, but the properties and advantages of such computations are still not well understood. In order to provide answers, in this work we demonstrate how Spiking neurons can enable temporal feature extraction in feed-forward neural networks without the need for recurrent synapses, and how recurrent SNNs can achieve comparable results to LSTM with a smaller number of parameters. This shows how their bio-inspired computing principles can be successfully exploited beyond energy efficiency gains and evidences their differences with respect to conventional artificial neural networks. These results are obtained through a new task, DVS-Gesture-Chain (DVS-GC), which allows, for the first time, to evaluate the perception of temporal dependencies in a real event-based action recognition dataset. Our study proves how the widely used DVS Gesture benchmark can be solved by networks without temporal feature extraction when its events are accumulated in frames, unlike the new DVS-GC which demands an understanding of the order in which events happen. Furthermore, this setup allowed us to reveal the role of the leakage rate in spiking neurons for temporal processing tasks and demonstrated the benefits of ”hard reset” mechanisms. Additionally, we also show how time-dependent weights and normalisation can lead to understanding order by means of temporal attention. Code for the DVS-GC task is available.

Lightweight monocular depth estimation using a fusion-improved transformer

The existing deep estimation networks often overlook the issue of computational efficiency while pursuing high accuracy. This paper proposes a lightweight self-supervised network that combines convolutional neural networks (CNN) and Transformers as the feature extraction and encoding layers for images, enabling the network to capture both local geometric and global semantic features for depth estimation. First, depth-separable convolution is used to construct a dilated convolution residual module based on a shallow network to improve the shallow CNN feature extraction receptive field. In the transformer, a multidepth separable convolution head transposed attention module is proposed to reduce the computational burden of spatial self-attention. In the feedforward network, a two-step gating mechanism is proposed to improve the nonlinear representation ability of the feedforward network. Finally, the CNN and transformer are integrated to implement a depth estimation network with a local-global context interaction function. Compared with other lightweight models, this model has fewer model parameters and higher estimation accuracy. It also has better generalizability for different outdoor datasets. Additionally, the inference speed can reach 87 FPS, achieving better real-time performance and accounting for both inference speed and estimation accuracy.