Feedforward Network Research Articles

Enhancing the analysis of murine neonatal ultrasonic vocalizations: Development, evaluation, and application of different mathematical models.

Rodents employ a broad spectrum of ultrasonic vocalizations (USVs) for social communication. As these vocalizations offer valuable insights into affective states, social interactions, and developmental stages of animals, various deep learning approaches have aimed at automating both the quantitative (detection) and qualitative (classification) analysis of USVs. So far, no notable efforts have been made to determine the most suitable architecture. We present the first systematic evaluation of different types of neural networks for USV classification. We assessed various feedforward networks, including a custom-built, fully-connected network, a custom-built convolutional neural network, several residual neural networks, an EfficientNet, and a Vision Transformer. Our analysis concluded that convolutional networks with residual connections specifically adapted to USV data, are the most suitable architecture for analyzing USVs. Paired with a refined, entropy-based detection algorithm (achieving recall of 94.9 % and precision of 99.3 %), the best architecture (achieving 86.79 % accuracy) was integrated into a fully automated pipeline capable of analyzing extensive USV datasets with high reliability. In ongoing projects, our pipeline has proven to be a valuable tool in studying neonatal USVs. By comparing these distinct deep learning architectures side by side, we have established a solid foundation for future research.

DT-Block: Adaptive vertical federated reinforcement learning scheme for secure and efficient communication in 6G

The necessities of security and data sharing have focused on federated learning because of using decentralized data sources. The existing works used federated learning for security, however, it still faces many challenges such as poor security and privacy, computational complexity, etc. In this research, we propose adaptive vertical federated learning using a reinforcement learning approach and blockchain. The proposed work includes three phases: user registration and authentication, machine learning-based client selection, and adaptive secure federated learning. Initially, all the users register their credentials to the cognitive agent, which generates a private key, public key, and random number using a Chaotic Isogenic Post Quantum Cryptography (CIPQC) algorithm. Second, optimal clients are selected for participating in federated learning which improves learning rate and reduces complexity. Here, optimal clients are selected by the Enhanced Multilayer Feed Forward Neural Network (EMFFN) algorithm by considering CSI, RSSI, bandwidth, energy, communication efficiency, and statistical efficiency. Finally, adaptive secure federated learning is performed by the Distributed Distributional Deep Deterministic Policy Gradient (D4PG) algorithm, where the local models are adaptively used by the private strategy based on its sensitivity. The aggregated global models are stored in DT-block (dendrimer tree-based blockchain) which stores the data in a dendrimer tree structure for increasing scalability and reducing search time during data retrieval. The simulation of this research is conducted by NS-3.26 network simulator and the performance of the proposed DT-Block model is estimated based on various performance metrics such as accuracy, delay, loss, f1-score, and security strength this demonstrated that the suggested effort produced better results both in terms of quantitative and qualitative aspects.

Analytical Bounds on the Local Lipschitz Constants of ReLU Networks.

In this article, we determine analytical upper bounds on the local Lipschitz constants of feedforward neural networks with rectified linear unit (ReLU) activation functions. We do so by deriving Lipschitz constants and bounds for ReLU, affine-ReLU, and max pooling functions, and combining the results to determine a network-wide bound. Our method uses several insights to obtain tight bounds, such as keeping track of the zero elements of each layer, and analyzing the composition of affine and ReLU functions. Furthermore, we employ a careful computational approach which allows us to apply our method to large networks such as AlexNet and VGG-16. We present several examples using different networks, which show how our local Lipschitz bounds are tighter than the global Lipschitz bounds. We also show how our method can be applied to provide adversarial bounds for classification networks. These results show that our method produces the largest known bounds on minimum adversarial perturbations for large networks such as AlexNet and VGG-16.

A Human Resource Evaluation and Recommendation System based on Big Data Mining

This investigation presents a paradigm-shifting Human Resource (HR) Assessment and Recommendation Framework, leveraging progressed huge information mining procedures. Drawing bits of knowledge from cybersecurity, healthcare, education, and further detecting spaces, our framework utilizes a different cluster of calculations, counting Arbitrary Forests, Support Vector Machines, K-Means Clustering, and a Feedforward Neural Organize. The comparative investigation uncovers the Feedforward Neural Network as the standout entertainer, emphasizing its various levels including learning for complicated design acknowledgement inside HR measurements. Uniquely, this framework draws on methodologies and insights from varied domains such as cybersecurity, healthcare, and education, applying these rich, interdisciplinary perspectives to HR analytics. This cross-pollination of ideas enables the framework to adopt sophisticated data mining and pattern recognition techniques that are not traditionally utilized within HR, offering new avenues for detecting and interpreting complex employee data patterns. Result values illustrate the system’s adequacy, with a precision of 88%, an accuracy of 90%, a review of 87%, and an F1 score of 88%. These measurements emphasize the system’s capacity to comprehensively assess worker execution, giving exact suggestions for key HR decision-making. Ethical contemplations, innovation acknowledgement, and custom-fitted proposal frameworks, propelled by related works, are coordinates to guarantee the system’s reasonability over assorted organizational settings. This research contributes to the advancing scene of HR administration, offering a spearheading arrangement for organizations looking for data-driven, comprehensive, and morally sound approaches to workforce optimization.

Probabilistic Estimation of Parameters for Lubrication Application with Neural Networks

This paper investigates the use of neural networks to predict characteristic parameters of the grease application process pressure curve. A combination of two feed-forward neural networks was used to estimate both the value and the standard deviation of selected features. Several neuron configurations were tested and evaluated in their capability to make a probabilistic estimation of the lubricant’s parameters. The value network was trained with a dataset containing the full set of features and with a dataset containing its average values. As expected, the full network was able to predict noisy features well, while the average network made smoother predictions. This is also represented by the networks’ R2 values which are 0.781 for the full network and 0.737 for the mean network. Several further neuron configurations were tested to find the smallest possible configuration. The analysis showed that three or more neurons deliver the best fit over all features, while one or two neurons are not sufficient for prediction. The results showed that the grease application process pressure curve via pressure valves can be estimated by using neural networks.

Assessment of sealing systems impact on the vibration and environmental safety of rotary machines

Energy saving control algorithms of centrifugal fans/pumps are based on the use of the frequency-controlled induction motor drives and pressure or flow rate sensors, the costs of which are comparable to the cost of the fans/pumps for low-power applications. The paper develops a new and simple estimation approach of the pressure and flow rate, utilising the measured Root Mean Square (RMS) value of the stator current, estimated motor’s input active power, reference stator voltage frequency and feed-forward backpropagation artificial neural network. The error percentage for both flow rate and pressure in experimental and estimated data is within the range of ±5%, which conforms to the ISO 13348 standard. A test rig for the rapid control prototyping of the fan is designed, and necessary design and test procedures are developed. The estimation approach is verified experimentally and demonstrates better estimation accuracy compared to the existing and possible similar simple approaches. The developed algorithm can be easily embedded into the industrial variable frequency drives without any hardware changes.

Improved Flow Rate and Pressure ANN Estimators for a Centrifugal Fan with Induction Motor Drive

Energy saving control algorithms of centrifugal fans/pumps are based on the use of the frequency-controlled induction motor drives and pressure or flow rate sensors, the costs of which are comparable to the cost of the fans/pumps for low-power applications. The paper develops a new and simple estimation approach of the pressure and flow rate, utilising the measured Root Mean Square (RMS) value of the stator current, estimated motor’s input active power, reference stator voltage frequency and feed-forward backpropagation artificial neural network. The error percentage for both flow rate and pressure in experimental and estimated data is within the range of ±5%, which conforms to the ISO 13348 standard. A test rig for the rapid control prototyping of the fan is designed, and necessary design and test procedures are developed. The estimation approach is verified experimentally and demonstrates better estimation accuracy compared to the existing and possible similar simple approaches. The developed algorithm can be easily embedded into the industrial variable frequency drives without any hardware changes.

Neural networks for ID gap orbit distortion compensation in PETRA III

Undulators are used in storage rings to produce extremely brilliant synchrotron radiation. In the ideal case, a perfectly tuned undulator always has a first and second field integrals equal to zero. But, in practice, field integral changes during gap movements can never be avoided for real-life devices. As they significantly impact the circulating electron beam, there is the need to routinely compensate such effects. Deep Neural Networks can be used to predict the distortion in the closed orbit induced by the undulator gap variations on the circulating electron beam. In this contribution several state-of-the-art deep learning algorithms were trained on measurements from PETRA III. The different architecture performances are then compared to identify the best model for the gap-induced distortion compensation. It is found that realistic data, with several gaps moving simultaneously to arbitrary gaps must be used to train the neural networks. The deep feed-forward neural network was found to be more effective than the recurrent and convolutional neural networks.

Email Spam Detection Using a Hybrid Approach of Feedforward Neural Network and Penguin Optimization Algorithm

The electronic communication of today is beset by unwanted junk messages. These are not just a nuisance; they waste our time and energy and clog up our inboxes, which are supposed to be, in many ways, our digital front doors. But how do we keep them from coming? Well, as today's article makes clear, the situation is hardly straightforward. There are various methods that can be used, both traditional and modern, and each has its own set of pros and cons. One method, however, stands out in today's article as one that we think might work well in practice, the good old-fashioned neural network. The method's novelty derives from its use of neural networks, a type of machine learning, in conjunction with the POA. The POA is a powerful optimization algorithm that can perform very well when fine-tuning the weights and biases of a neural network. We therefore used the POA to optimize these two crucial components of our model and achieved a surprising level of accuracy—98.7%. This figure surpasses the next nearest competitor by 2% and demonstrates the efficacy of our method in spam detection while also highlighting the POA's potential in this field. The overall proposed method pushes the spam-detection field forward, but it also demonstrates something even broader: the power of hybrid approaches in machine learning. Neural networks are great—but you have to train them properly, and the use of a particle swarm for that purpose is a neat idea. But in the end, a hybrid spam filter is still a spam filter, and this one does what it's supposed to do better than many rivaling methods.

Spiking Neural Networks for event-based action recognition: A new task to understand their advantage

Spiking Neural Networks (SNN) are characterised by their unique temporal dynamics, but the properties and advantages of such computations are still not well understood. In order to provide answers, in this work we demonstrate how Spiking neurons can enable temporal feature extraction in feed-forward neural networks without the need for recurrent synapses, and how recurrent SNNs can achieve comparable results to LSTM with a smaller number of parameters. This shows how their bio-inspired computing principles can be successfully exploited beyond energy efficiency gains and evidences their differences with respect to conventional artificial neural networks. These results are obtained through a new task, DVS-Gesture-Chain (DVS-GC), which allows, for the first time, to evaluate the perception of temporal dependencies in a real event-based action recognition dataset. Our study proves how the widely used DVS Gesture benchmark can be solved by networks without temporal feature extraction when its events are accumulated in frames, unlike the new DVS-GC which demands an understanding of the order in which events happen. Furthermore, this setup allowed us to reveal the role of the leakage rate in spiking neurons for temporal processing tasks and demonstrated the benefits of ”hard reset” mechanisms. Additionally, we also show how time-dependent weights and normalisation can lead to understanding order by means of temporal attention. Code for the DVS-GC task is available.

Lightweight monocular depth estimation using a fusion-improved transformer

The existing deep estimation networks often overlook the issue of computational efficiency while pursuing high accuracy. This paper proposes a lightweight self-supervised network that combines convolutional neural networks (CNN) and Transformers as the feature extraction and encoding layers for images, enabling the network to capture both local geometric and global semantic features for depth estimation. First, depth-separable convolution is used to construct a dilated convolution residual module based on a shallow network to improve the shallow CNN feature extraction receptive field. In the transformer, a multidepth separable convolution head transposed attention module is proposed to reduce the computational burden of spatial self-attention. In the feedforward network, a two-step gating mechanism is proposed to improve the nonlinear representation ability of the feedforward network. Finally, the CNN and transformer are integrated to implement a depth estimation network with a local-global context interaction function. Compared with other lightweight models, this model has fewer model parameters and higher estimation accuracy. It also has better generalizability for different outdoor datasets. Additionally, the inference speed can reach 87 FPS, achieving better real-time performance and accounting for both inference speed and estimation accuracy.

Deep learning for solving initial path optimization of mean-field systems with memory

We consider the problem of finding the optimal initial investment strategy for a system modelled by a linear McKean–Vlasov (mean-field) stochastic differential equation with delay, driven by Brownian motion and a pure jump Poisson random measure. The goal is to determine the optimal initial values for the system in the period [ − δ , 0 ] , where δ > 0 is a delay constant, before the system starts at t = 0. Due to the delay in the dynamics, the system will, after startup, be influenced by these initial investment values. It is known that linear stochastic delay differential equations are equivalent to stochastic Volterra integral equations. By utilizing this equivalence, we can find implicit expressions for the optimal investment. Moreover, we propose a deep neural network-based algorithm to solve the stochastic control problem with delay. Specifically, we employ a multi-layer feed-forward neural network for control modelling in the interval [ − δ , 0 ] , and use back-propagation to train the feed-forward neural network. The gradient of the loss function is computed using stochastic gradient descent (SGD) with respect to the weights of the network.

Deep learning for blood glucose level prediction: How well do models generalize across different data sets?

Deep learning-based models for predicting blood glucose levels in diabetic patients can facilitate proactive measures to prevent critical events and are essential for closed-loop control therapy systems. However, selecting appropriate models from the literature may not always yield conclusive results, as the choice could be influenced by biases or misleading evaluations stemming from different methodologies, datasets, and preprocessing techniques. This study aims to compare and comprehensively analyze the performance of various deep learning models across diverse datasets to assess their applicability and generalizability across a broader spectrum of scenarios. Commonly used deep learning models for blood glucose level forecasting, such as feed-forward neural network, convolutional neural network, long short-term memory network (LSTM), temporal convolutional neural network, and self-attention network (SAN), are considered in this study. To evaluate the generalization capabilities of each model, four datasets of varying sizes, encompassing samples from different age groups and conditions, are utilized. Performance metrics include Root Mean Square Error (RMSE), Mean Absolute Difference (MAD), and Coefficient of Determination (CoD) for analytical asssessment, Clarke Error Grid (CEG) for clinical assessments, Kolmogorov-Smirnov (KS) test for statistical analysis, and generalization ability evaluations to obtain both coarse and granular insights. The experimental findings indicate that the LSTM model demonstrates superior performance with the lowest root mean square error and highest generalization capability among all other models, closely followed by SAN. The ability of LSTM and SAN to capture long-term dependencies in blood glucose data and their correlations with various influencing factors and events contribute to their enhanced performance. Despite the lower predictive performance, the FFN was able to capture patterns and trends in the data, suggesting its applicability in forecasting future direction. Moreover, this study helps in identifying the optimal model based on specific objectives, whether prioritizing generalization or accuracy.

Novel spatio-temporal attention causal convolutional neural network for multi-site PM2.5 prediction

Multi-site PM2.5 prediction has emerged as a crucial approach, given that the accuracy of prediction models based solely on data from a single monitoring station may be constrained. However, existing multi-site PM2.5 prediction methods predominantly rely on recurrent networks for extracting temporal dependencies and overlook the domain knowledge related to air quality pollutant dispersion. This study aims to explore whether a superior prediction architecture exists that not only approximates the prediction performance of recurrent networks through feedforward networks but also integrates domain knowledge of PM2.5. Consequently, we propose a novel spatio-temporal attention causal convolutional neural network (Causal-STAN) architecture for predicting PM2.5 concentrations at multiple sites in the Yangtze River Delta region of China. Causal-STAN comprises two components: a multi-site spatio-temporal feature integration module, which identifies temporal local correlation trends and spatial correlations in the spatio-temporal data, and extracts inter-site PM2.5 concentrations from the directional residual block to delineate directional features of PM2.5 concentration dispersion between sites; and a temporal causal attention convolutional network that captures the internal correlation information and long-term dependencies in the time series. Causal-STAN was evaluated using one-year data from 247 sites in mainland China. Compared to six state-of-the-art baseline models, Causal-STAN achieves optimal performance in 6-hour future predictions, surpassing the recurrent network model and reducing the prediction error by 8%–10%.

Road surface semantic segmentation for autonomous driving

Although semantic segmentation is widely employed in autonomous driving, its performance in segmenting road surfaces falls short in complex traffic environments. This study proposes a frequency-based semantic segmentation with a transformer (FSSFormer) based on the sensitivity of semantic segmentation to frequency information. Specifically, we propose a weight-sharing factorized attention to select important frequency features that can improve the segmentation performance of overlapping targets. Moreover, to address boundary information loss, we used a cross-attention method combining spatial and frequency features to obtain further detailed pixel information. To improve the segmentation accuracy in complex road scenarios, we adopted a parallel-gated feedforward network segmentation method to encode the position information. Extensive experiments demonstrate that the mIoU of FSSFormer increased by 2% compared with existing segmentation methods on the Cityscapes dataset.

Machine Learning and Deep Learning Models for Predicting Noncovalent Inhibitors of AmpC β-Lactamase.

Continuous use of antibiotics leads to the ability of bacteria to adapt by developing complex antibiotic resistance (AR) mechanisms. The synthesis of β-lactamases is a widely observed AR mechanism. The class C β-lactamase (AmpC) causes significant resistance toward β-lactam antibiotics, and new treatments are urgently needed. Noncovalent inhibitors have been developed against a broad spectrum of β-lactamases. In this study, we developed robust and accurate models for predicting noncovalent inhibitors of AmpC using large compound data sets and machine/deep learning modeling. We created support vector machine (SVM), random forest (RF), and feed-forward neural network (FFNN) classification models. The cross-validation (CV) accuracies varied between 80 and 82%, as combined models reached an accuracy of 83%. We analyzed the physicochemical characteristics of the noncovalent inhibitors and predicted the binding modes for some of them. Such models are helpful for identifying new noncovalent inhibitors in order to establish novel solutions against the growing resistance to standard β-lactam inhibitors. The best RF, SVM, and FFNN models for predicting noncovalent inhibitors of AmpC β-lactamase are available in the GitHub repository, https://github.com/UPCmctr/ML-DL-AmpC-B-lactamase.

Implementation of a multistage predictive energy management strategy considering electric vehicles using a novel hybrid optimization technique

In this paper, a novel artificial intelligence (AI)-based energy management strategy across three levels is designed for isolated microgrids. During the initial phase, AI is not employed. A precise and rapid-response microcontroller, namely the FPGA, is utilized. The performance of the FPGA is experimentally investigated under various operation conditions. In the second phase, AI plays a crucial role in enhancing both the reliability and economic effectiveness of the system. The multi-objective snake optimization (SO) algorithm is employed to attain high technical and economic performance within predefined constraints. Three objective functions are involved in this phase, focusing on the minimization of operational costs, loss of power supply probability (LPSP), and electrical energy losses in the dummy load. In the third level, a novel control strategy-based coordinated model predictive control is presented as part of the proposed AI-embedded energy management strategy. A hybrid optimization algorithm combining the multilayer feed-forward neural networks (MFFNN) and SO algorithms is proposed to predict the output power of backup sources. The microgrid under consideration is supported by a hybrid backup system comprising battery energy storage systems (BESS), electric vehicle (EV) batteries, and fuel cells (FCs). The effectiveness of this hybrid algorithm is evaluated through comparison with many other algorithms, aiming to assess its performance in accurately predicting the output power of backup sources. The results show the feasibility of using AI to achieve efficient operation and energy management in microgrids. The proposed MFFNN-SO algorithm achieves the best performance, as the normalized root mean square error (NRMSE) for FCs, BESS, and EV is about 0.1296 %, 0.0094 %, and 0.1304 %, respectively. The SO algorithm achieves a notable 6.3361% reduction in operating costs, resulting in a final operational cost of 166.4811 $.

Optimizing High-Performance Predictive Modeling of the Medium-Speed WEDM Processing of Inconel 718

The purpose of this research was to create a predictive model for a medium-speed wire electrical discharge machine (WEDM) utilizing an artificial neural network (ANN). Medium-speed WEDM experiments were developed based on the I-optimal mixture design for machining, the Inconel 718 superalloy. During the experiment, the input parameters were the spark ontime, spark offtime, wire feed, and current, with the material removal rate (MRR) and surface roughness (Ra) selected as performance indicators. The ANN model was trained on experimental data and built using a feed-forward backpropagation neural network with a (4-8-2) structure and the Bayesian regularization (BR) learning approach. The model correctly predicted the relationship between the medium-speed WEDM’s primary process parameters and machining performance. An integrated ANN model and the Non-Dominated Sorting Genetic Algorithm-II (NSGA-II) were used to determine the ideal parameters for the MRR and Ra, resulting in a set of Pareto-optimal solutions. The confirmation experiment revealed that the mean prediction error between the experimental and ideal solutions had a maximum error percentage of 1% for the MRR and 2% for the Ra, which are within acceptable ranges. This showed that the best process–parameter combinations were better for the MRR and Ra.

Identification of Distorted Gamma-Ray Signature Patterns Using Digital Filtering and Auto-Associative Memory Implemented with a Hopfield Neural Network

The detection and identification of radioactive sources in search applications involve analyzing passive gamma-ray emissions from high-level radioactive materials. This process uses a mobile detector-spectrometer in a complex field test environment. Recently, the use of artificial intelligence for gamma-ray spectrum analysis has shown promising results. However, challenges persist in identifying isotopic signatures from spectral measurements that may be distorted due to source shielding, random variations in natural radioactive background, or insufficient measurement time to obtain clear spectral lines. This paper presents a novel intelligent signature recognition method that combines digital filtering techniques with an artificial Hopfield Neural Network (HNN). The HNN leverages auto-associative memory to store training sample patterns and match them with incoming gamma spectra from distorted sources. It restores the testing sources’ measurements by finding the closest matching signature patterns in the spectral library. Before HNN recognition, the measured spectrum undergoes preprocessing with a digital image filter to reduce fluctuations. Performance of the proposed method is evaluated using a set of gamma-ray spectra measured with a sodium iodide detector. The data collected include measurements from six pure samples: 241Am, 60Co, 137Cs, 192Ir, 239Pu, and 235U, which are used for training and validation (i.e. six cases). Additionally, the data set contains 24 distorted synthesized sources with various fluctuating backgrounds. Test results demonstrate the potential of the proposed method to accurately recognize the correct isotope with high precision, achieving an accuracy rate exceeding 85%. Furthermore, the proposed method exhibits superior performance compared to the conventional multiple regression fitting and simple feedforward neural network methods.

Leveraging feed-forward neural networks to enhance the hybrid block derivative methods for system of second-order ordinary differential equations

This study introduces an innovative method combining discrete hybrid block techniques and artificial intelligence to enhance the solution of second-order Ordinary Differential Equations (ODEs). By integrating feed-forward neural networks (FFNN) into the hybrid block derivative method (HBDM), the modified approach shows improved accuracy and efficiency compared to traditional methods. Through comprehensive comparisons with exact and existing solutions, the study demonstrates the effectiveness of the proposed approach. The evaluation, utilizing root mean square error (RMSE), confirms its superior performance, robustness, and applicability in diverse scenarios. This research sets a new standard for solving complex ODE systems, offering promising avenues for future research and practical implementations.