Training Feedforward Neural Networks Research Articles

Automated prediction of phosphorus concentration in soils using reflectance spectroscopy and machine learning algorithms

A method is presented for predicting total phosphorus concentration in soils from Santander de Quilichao, Colombia, using a UV-VIS V-750 Spectrophotometer and machine learning techniques. A total of 152 soil samples, prepared with varying proportions of P2O5 fertilizer and soil, were analyzed, obtaining reflectance spectra in the 200 to 900 nm range with 3501 wavelengths. Additionally, 152 laboratory results of total phosphorus concentration were used to train the prediction model. The spectra were filtered using a Savitzky-Golay filter. Key wavelengths were identified using Variable Importance in Projection - Partial Least Squares (VIP-PLS) and Random Forest (RF), reducing the spectral bands to 1085. Principal Component Analysis (PCA) further reduced data dimensionality. A feedforward artificial neural network was then trained to predict phosphorus concentration. This method is faster than traditional lab tests by leveraging advanced data analysis and machine learning, offering results in less time. While sample preparation remains consistent with standard spectroscopic analysis, the value added by the proposed method lies in its data processing and interpretation. Currently applied to a single soil type, future improvements will include more soil types and other macronutrients, enhancing nutrient management in agriculture. Accurate macronutrient measurements aid in better fertilizer uses planning.• Filtering spectra and determining relevant wavelengths using VIP-PLS and RF.• Dimensionality reduction with PCA.• Training feedforward artificial neural networks.

Network anomaly detection using Deep Autoencoder and parallel Artificial Bee Colony algorithm-trained neural network

Cyberattacks are increasingly becoming more complex, which makes intrusion detection extremely difficult. Several intrusion detection approaches have been developed in the literature and utilized to tackle computer security intrusions. Implementing machine learning and deep learning models for network intrusion detection has been a topic of active research in cybersecurity. In this study, artificial neural networks (ANNs), a type of machine learning algorithm, are employed to determine optimal network weight sets during the training phase. Conventional training algorithms, such as back-propagation, may encounter challenges in optimization due to being entrapped within local minima during the iterative optimization process; global search strategies can be slow at locating global minima, and they may suffer from a low detection rate. In the ANN training, the Artificial Bee Colony (ABC) algorithm enables the avoidance of local minimum solutions by conducting a high-performance search in the solution space but it needs some modifications. To address these challenges, this work suggests a Deep Autoencoder (DAE)-based, vectorized, and parallelized ABC algorithm for training feed-forward artificial neural networks, which is tested on the UNSW-NB15 and NF-UNSW-NB15-v2 datasets. Our experimental results demonstrate that the proposed DAE-based parallel ABC-ANN outperforms existing metaheuristics, showing notable improvements in network intrusion detection. The experimental results reveal a notable improvement in network intrusion detection through this proposed approach, exhibiting an increase in detection rate (DR) by 0.76 to 0.81 and a reduction in false alarm rate (FAR) by 0.016 to 0.005 compared to the ANN-BP algorithm on the UNSW-NB15 dataset. Furthermore, there is a reduction in FAR by 0.006 to 0.0003 compared to the ANN-BP algorithm on the NF-UNSW-NB15-v2 dataset. These findings underscore the effectiveness of our proposed approach in enhancing network security against network intrusions.

GSK-LocS: Towards a more effective generalisation in population-based neural network training

Despite the effectiveness of deep neural networks, feed-forward neural networks (FFNNs) continue to play a crucial role in many applications, especially when dealing with limited data availability. The primary challenge in FFNNs is determining the optimal weights during the training process, aiming to minimise the disparity between actual and predicted outputs. Although gradient-based techniques like backpropagation (BP) have traditionally been popular for FFNN training, they come with inherent limitations, such as sensitivity to initial weights and susceptibility to getting trapped in local optima. To overcome these challenges, we introduce a novel approach based on the Gaining-Sharing Knowledge-based(GSK) algorithm. To the best of our knowledge, this paper represents the first exploration of GSK for neural network training. After obtaining the appropriate weights for the FFNN by the GSK, the weights and biases are utilised to initialise a Levenberg–Marquardt backpropagation (LMBP) algorithm, serving as a local search component. In other words, our proposed algorithm, GSK-LocS, leverages the global search capabilities of the GSK algorithm and combines them with the local search capabilities of LMBP for neural network training. This integration mitigates sensitivity to initial values and reduces the risk of being trapped in local optima. Experimental results conducted on classification and approximation problems provide compelling evidence that our proposed algorithm is highly competitive compared to other existing methods.

Training feedforward neural networks with Bayesian hyper-heuristics

The process of training feedforward neural networks (FFNNs) can benefit from an automated process where the best heuristic to train the network is sought out automatically by means of a high-level probabilistic-based heuristic. This research introduces a novel population-based Bayesian hyper-heuristic (BHH) that is used to train feedforward neural networks (FFNNs). The performance of the BHH is compared to that of ten popular low-level heuristics, each with different search behaviours. The chosen heuristic pool consists of classic gradient-based heuristics as well as meta-heuristics (MHs). The empirical process is executed on fourteen datasets consisting of classification and regression problems with varying characteristics. The BHH is shown to be able to train FFNNs well and provide an automated method for finding the best heuristic to train the FFNNs at various stages of the training process.

An Enhanced Tunicate Swarm Algorithm with Symmetric Cooperative Swarms for Training Feedforward Neural Networks

The input layer, hidden layer, and output layer are three models of neural processors that comprise feedforward neural networks. In this paper, an enhanced tunicate swarm algorithm based on a differential sequencing alteration operator (ETSA) with symmetric cooperative swarms is presented to train feedforward neural networks. The objective is to accomplish minimum classification errors and the most appropriate neural network layout by regulating the layers’ connection weights and neurons’ deviation thresholds according to the transmission error between the anticipated input and the authentic output. The TSA mimics jet motorization and swarm scavenging to mitigate directional collisions and to maintain the greatest solution that is customized and regional. However, the TSA exhibits the disadvantages of low computational accuracy, a slow convergence speed, and easy search stagnation. The differential sequencing alteration operator has adaptable localized extraction and search screening to broaden the identification scope, enrich population creativity, expedite computation productivity, and avoid search stagnation. The ETSA integrates exploration and exploitation to mitigate search stagnation, which has sufficient stability and flexibility to acquire the finest solution. The ETSA was distinguished from the ETTAO, EPSA, SABO, SAO, EWWPA, YDSE, and TSA by monitoring seventeen alternative datasets. The experimental results confirm that the ETSA maintains profound sustainability and durability to avoid exaggerated convergence, locate the acceptable transmission error, and equalize extraction and prospection to yield a faster convergence speed, superior calculation accuracy, and greater categorization accuracy.

NNBGWO-BRCA marker: Neural Network and binary grey wolf optimization based Breast cancer biomarker discovery framework using multi-omics dataset

Background and ObjectiveBreast cancer is a multifaceted condition characterized by diverse features and a substantial mortality rate, underscoring the imperative for timely detection and intervention. The utilization of multi-omics data has gained significant traction in recent years to identify biomarkers and classify subtypes in breast cancer. This kind of research idea from part to whole will also be an inevitable trend in future life science research. Deep learning can integrate and analyze multi-omics data to predict cancer subtypes, which can further drive targeted therapies. However, there are few articles leveraging the nature of deep learning for feature selection. Therefore, this paper proposes a Neural Network and Binary grey Wolf Optimization based BReast CAncer bioMarker (NNBGWO-BRCAMarker) discovery framework using multi-omics data to obtain a series of biomarkers for precise classification of breast cancer subtypes. MethodsNNBGWO-BRCAMarker consists of two phases: in the first phase, relevant genes are selected using the weights obtained from a trained feedforward neural network; in the second phase, the binary grey wolf optimization algorithm is leveraged to further screen the selected genes, resulting in a set of potential breast cancer biomarkers. ResultsThe SVM classifier with RBF kernel achieved a classification accuracy of 0.9242 ± 0.03 when trained using the 80 biomarkers identified by NNBGWO-BRCAMarker, as evidenced by the experimental results. We conducted a comprehensive gene set analysis, prognostic analysis, and druggability analysis, unveiling 25 druggable genes, 16 enriched pathways strongly linked to specific subtypes of breast cancer, and 8 genes linked to prognostic outcomes. ConclusionsThe proposed framework successfully identified 80 biomarkers from the multi-omics data, enabling accurate classification of breast cancer subtypes. This discovery may offer novel insights for clinicians to pursue in further studies.

Intelligent PID Controller Based on Neural Network for AI-Driven Control Quadcopter UAV

Unmanned Aerial Vehicle (UAV), specifically a quadcopter is publicly popular which it provides services in different applications such as aerial delivery, aerial photography, military, weather forecasting and more examples to date. A Proportional-Integral-Derivative (PID) controller is one of the control techniques that can provide stabilization and reliable trajectory tracking. However, proper PID gains are needed to ensure a stable flight and it should be hybridized or improved to increase the robustness, reliability, and stabilization during flight. In this paper, an intelligent PID controller using neural network is proposed based on Levenberg-Marquardt feedforward neural network training method. The PID gains are initialized using different ranges according to the optimal gains generated by Particle Swarm Optimization, and this contributes towards a good training performance using Mean Square Error (MSE) evaluation. The trained network takes desired output and references as input data to calculate the required combination of PID gains as the output. The including of the response characteristics as the input data for the network, together with reference, error, and control input is the significance of the work. The performance of this work is presented using MSE performances, attitudes and altitude stabilization, and trajectory tracking reliability through error index performances. The simulation results graphically prove that the proposed controller provides better stability with reduced overshoot and settling times. Disturbance rejection is also enhanced by 1.7% compared to manual tuned PID controller. The reliability of the proposed controller highlights avenues for further exploration in AI-driven control strategies for quadcopter systems.

Thermo-magnetic radiative flow in porous enclosure with deep-learning parameter estimation

This study numerically investigates convective heat transfer, fluid flow, and entropy generation within an inverted T-shaped porous enclosure filled with a hybrid nanofluid. The interplay between thermal radiation and a magnetic field is considered, as these factors significantly influence thermal acquisition processes in renewable and thermal engineering applications, particularly solar power collectors with the selected physical domain. By analyzing various combinations of flow parameters, this work aims to identify favorable conditions for maximizing heat transfer efficiency in such systems. A Darcy–Forchheimer–Brinkman-based mathematical model has been incorporated and numerically simulated using the penalty finite element method (FEM). Moreover, the variation of fluid flow, isotherms, and entropy generation is thoroughly analyzed for the variation of non-dimensional permeability parameter (Darcy number, Da), buoyancy force (Rayleigh number, Ra), magnetic field (Hartmann number, Ha), porosity value (ϵ), and thermal radiation parameter (Rd). The comprehensive result demonstrates that increasing values of Ra, Da, ϵ, and Rd intensify entropy generation, convective heat, and fluid flow phenomena, whereas the increasing value of Ha substantially weakens these phenomena. The thermal efficiency of the model attends at the lower buoyancy forces (Ra≤105) for each flow parameter (except for Ha). However, the increasing magnetic forces (Ha) substantially improve the thermal efficiency of the developed model. Additionally, this study employs a feed-forward neural network (FNN) based deep learning (DL) approach to predict the parametric influence on convective heat transport rate (Num), total entropy generation rate (Stot), and total Bejan number due to heat transport irreversibility (Beθ,tot) based on the minimally supplied experimental data for FNN training process. The trained FNN model predicts the results very quickly with 98.6% accuracy compared to FEM results, demonstrating its efficiency in approximating the parametric influence of complex mathematical model results. Moreover, this approach significantly reduces computational costs and efforts compared to traditional numerical methods, making it a valuable tool for forecasting the influence of an unknown set of flow parameters in complex mathematical models.

A Hybrid Neural Network‐Based Improved PSO Algorithm for Gas Turbine Emissions Prediction

AbstractIn gas‐fired power plants, emissions may reduce turbine blade rotation, thus decreasing power output. This study proposes a hybrid model integrating the Feed forward Neural Network (FFNN) model and Particle Swarm Optimization (PSO) algorithm to predict gas emissions from natural gas power plants. The FFNN predicts gas turbine nitrogen oxides (NOx) and carbon monoxide (CO) emissions, while the PSO optimizes FFNN weights, improving prediction accuracy. The PSO adopts a unique random number selection strategy, incorporating the K‐Nearest Neighbor (KNN) algorithm to reduce prediction errors. Neighbor Component Analysis (NCA) selects parameters most correlated with CO and NOx emissions. The hybrid model is constructed, trained, and testedusing publicly available datasets, evaluating performance with statistical metrics like Mean Square Error (MSE), Mean Absolute Error (MAE), and Root Mean Square Error (RMSE). Results show significant improvement in FFNN training with the PSO algorithm, boosting CO and NOx prediction accuracy by 99.18% and 82.11%, respectively. The model achieves the lowest MSE, MAE, and RMSE values for CO and NOx emissions. Overall, the hybrid model achieves high prediction accuracy, particularly with optimized PSO parameter selection using seed random generators.

Numerically Optimized Fourier Transform-Based Beamforming Accelerated by Neural Networks

Conventional beamforming methods for reconfigurable reflector antennas assume full control over the amplitude and phase of the reflected field. Here, we develop a novel beamforming methodology for reflecting Programmable Metasurfaces (PMS) with capacitive memory. Although utilizing such fully reactive PMS simplifies antenna design and reduces energy consumption, the PMS reflection magnitude is unity and thus a global optimization of the reflection phases over the PMS unit cells is required in each beamforming scenario. We propose an implementation of such an optimization method rooted in the traditional Fourier transform-based beamforming and evaluate its performance. Additionally, we show that a pair of trained feed-forward neural networks (FFNN) with one input, one hidden, and one output layer can replace time-consuming global optimizations in the case of a PMS comprising 3×10 unit cells. We train the FFNNs on a dataset obtained for typical single- and dual-beam beamforming scenarios. After training, the FFNNs perform requested beamforming tasks within a fraction of second and with about the same accuracy as the original optimization algorithm. The proposed methodology may find applications in future mobile telecommunication systems that require real-time beamforming on low-end hardware. The same beamforming methodology can be also employed in short-range wireless power transfer systems.

A NEURAL NETWORK PREDICTION MODEL FOR SURFACEFINISH IN TURNING PROCESS

This paper presents a neural network based surface finish Prediction model in turning operation .Orthogonal cutting tests were performed on mild steel using H.S.S cutting tool with differentcutting parameters cutting speed ,feed and nose radius of the cutting tool . The collected data wasused to train feed forward back propagation neural network. The developed model has been testedto preclict surface finish for various cutting conditions. The model was found to be powerful &capable of accurate surface finish prediction for the range it had been trained but the accuracydeteriorated as the cutting conditions were changed significantly'

Experimental Results of Underwater Sound Speed Profile Inversion by Few-Shot Multi-Task Learning

Underwater Sound Speed Profile (SSP) distribution is crucial for the propagation mode of acoustic signals, so fast and accurate estimation of SSP is of great importance in building underwater observation systems. The state-of-the-art SSP inversion methods include frameworks of matched field processing (MFP), compressive sensing (CS), and feed-forward neural networks (FNNs), among which the FNN shows better real-time performance while maintaining the same level of accuracy. However, the training of FNN needs quite a lot historical SSP samples, which is difficult to satisfy in many ocean areas. This situation is called few-shot learning. To tackle this issue, we propose a multi-task learning (MTL) model with partial parameter sharing among different training tasks. By MTL, common features could be extracted, which accelerates the learning process on given tasks, and reduces the demand for reference samples, enhancing the generalization ability in few-shot learning. To verify the feasibility and effectiveness of MTL, a deep-ocean experiment was held in April 2023 in the South China Sea. Results show that MTL outperforms the other mainstream methods in terms of accuracy for SSP inversion, while inheriting the real-time advantage of FNN during the inversion stage.

Convergence Analysis and Application for Multi‐Layer Neural Network Based on Fractional‐Order Gradient Descent Learning

AbstractFractional order calculus, with its inheritance and infinite memory properties, is a promising research area in information processing and modeling of certain physical systems, system identification, and control. In this paper, a fractional‐order gradient descent method is proposed for backpropagation training of multilayer feedforward neural networks. In particular, the Caputo derivative is used to define the measurement function and consider the smooth regular term. In addition, the monotonicity of the error function and the strong (weak) convergence theorem of the algorithm are rigorously proved. Finally, numerical experiments prove the correctness and effectiveness of the algorithm theory.

Grey Wolf Cuckoo Search Algorithm for Training Feedforward Neural Network and Logic Gates Design

This paper presents a new hybrid Swarm Intelligence (SI) algorithm based on the Cuckoo Search Algorithm (CSA) and Grey Wolf Optimizer (GWO) called the Grey Wolf Cuckoo Search (GWCS) algorithm. The GWCS algorithm extracts and combines CSA and GWO features for efficient optimization. To carry out the comprehensive validation, the developed algorithm is applied to three different scenarios with their counterparts. The first validation is carried out on standard optimization benchmark problems. Further, they are used to train Feedforward Neural Networks and finally applied to design logic gates. The comprehensive results are presented and it is found that the proposed GWCS algorithms perform better compared to the state-of-the-art.

Comparison of Swarm-based Metaheuristic and Gradient Descent-based Algorithms in Artificial Neural Network Training

This paper aims to compare the gradient descent-based algorithms under classical training model and swarm-based metaheuristic algorithms in feed forward backpropagation artificial neural network training. Batch weight and bias rule, Bayesian regularization, cyclical weight and bias rule and Levenberg-Marquardt algorithms are used as the classical gradient descent-based algorithms. In terms of the swarm-based metaheuristic algorithms, hunger games search, gray wolf optimizer, Archimedes optimization, and the Aquila optimizer are adopted. The Iris data set is used in this paper for the training. Mean square error, mean absolute error and determination coefficient are used as statistical measurement techniques to determine the effect of the network architecture and the adopted training algorithm. The metaheuristic algorithms are shown to have superior capability over the gradient descent-based algorithms in terms of artificial neural network training. In addition to their success in error rates, the classification capabilities of the metaheuristic algorithms are also observed to be in the range of 94%-97%. The hunger games search algorithm is also observed for its specific advantages amongst the metaheuristic algorithms as it maintains good performance in terms of classification ability and other statistical measurements.

Training of Feed-Forward Neural Networks by Using Optimization Algorithms Based on Swarm-Intelligent for Maximum Power Point Tracking.

One of the most used artificial intelligence techniques for maximum power point tracking is artificial neural networks. In order to achieve successful results in maximum power point tracking, the training process of artificial neural networks is important. Metaheuristic algorithms are used extensively in the literature for neural network training. An important group of metaheuristic algorithms is swarm-intelligent-based optimization algorithms. In this study, feed-forward neural network training is carried out for maximum power point tracking by using 13 swarm-intelligent-based optimization algorithms. These algorithms are artificial bee colony, butterfly optimization, cuckoo search, chicken swarm optimization, dragonfly algorithm, firefly algorithm, grasshopper optimization algorithm, krill herd algorithm, particle swarm optimization, salp swarm algorithm, selfish herd optimizer, tunicate swarm algorithm, and tuna swarm optimization. Mean squared error is used as the error metric, and the performances of the algorithms in different network structures are evaluated. Considering the results, a success ranking score is obtained for each algorithm. The three most successful algorithms in both training and testing processes are the firefly algorithm, selfish herd optimizer, and grasshopper optimization algorithm, respectively. The training error values obtained with these algorithms are 4.5 × 10-4, 1.6 × 10-3, and 2.3 × 10-3, respectively. The test error values are 4.6 × 10-4, 1.6 × 10-3, and 2.4 × 10-3, respectively. With these algorithms, effective results have been achieved in a low number of evaluations. In addition to these three algorithms, other algorithms have also achieved mostly acceptable results. This shows that the related algorithms are generally successful ANFIS training algorithms for maximum power point tracking.

On Performance of Marine Predators Algorithm in Training of Feed-Forward Neural Network for Identification of Nonlinear Systems

Artificial neural networks (ANNs) are used to solve many problems, such as modeling, identification, prediction, and classification. The success of ANN is directly related to the training process. Meta-heuristic algorithms are used extensively for ANN training. Within the scope of this study, a feed-forward artificial neural network (FFNN) is trained using the marine predators algorithm (MPA), one of the current meta-heuristic algorithms. Namely, this study is aimed to evaluate the performance of MPA in ANN training in detail. Identification/modeling of nonlinear systems is chosen as the problem. Six nonlinear systems are used in the applications. Some of them are static, and some are dynamic. Mean squared error (MSE) is utilized as the error metric. Effective training and testing results were obtained using MPA. The best mean error values obtained for six nonlinear systems are 2.3 × 10−4, 1.8 × 10−3, 1.0 × 10−4, 1.0 × 10−4, 1.2 × 10−5, and 2.5 × 10−4. The performance of MPA is compared with 16 meta-heuristic algorithms. The results have shown that the performance of MPA is better than other algorithms in ANN training for the identification of nonlinear systems.

Modified Backpropagation Algorithm with Multiplicative Calculus in Neural Networks

Backpropagation is one of the most widely used algorithms for training feedforward deep neural networks. The algorithm requires a differentiable activation function and it performs computations of the gradient proceeding backwards through the feedforward deep neural network from the last layer through to the first layer. In order to calculate the gradient at a specific layer, the gradients of all layers are combined via the chain rule of calculus. One of the biggest disadvantages of the backpropagation is that it requires a large amount of training time. To overcome this issue, this paper proposes a modified backpropagation algorithm with multiplicative calculus. Multiplicative calculus provides an alternative to the classical calculus and it defines new kinds of derivative and integral forms in multiplicative form rather than addition and subtraction forms. The performance analyzes are discussed in various case studies and the results are given comparatively with classical backpropagation algorithm. It is found that the proposed modified backpropagation algorithm converges in less time to the solution and thus provides fast training in the given case studies. It is also shown that the proposed algorithm avoids the local minima problem.

APPSO‐NN: An adaptive‐probability particle swarm optimization neural network for sensorineural hearing loss detection

AbstractAs a hearing disorder, sensorineural hearing loss (SNHL) can be effectively detected by magnetic resonance imaging (MRI). However, the manual detection of MRI scanning is subjective, time‐consuming, and unpredictable. An accurate and automatic computer‐aided diagnosis system is proposed for SNHL detection, providing reliable references for professionals. The system first employs a wavelet entropy layer to extract features of MRI images. Then, a neural network layer is proposed as the classifier consisting of a feedforward neural network (FNN) and an adaptive‐probability PSO (APPSO) algorithm. The authors prove the rotation‐variant property of the basic particle swarm optimization (PSO) by the algebraic property of matrix transformation. The property is unsuitable for optimising parameters of neural networks. Thus, in APPSO, the authors integrate the new update rules based on all‐dimensional variation and adaptive‐probability mechanism into the basic PSO, which can improve its searching ability without losing population diversity. The authors compare APPSO‐NN with FNN trained by five popular evolutionary algorithms. The simulation results show that APPSO performs best in training FNN. The method also compares with six state‐of‐the‐art methods. The simulation results show that the best performance in sensitivity and overall accuracy of hearing loss classification, which proves that the method is effective and promising for SNHL detection.

A lifted Bregman formulation for the inversion of deep neural networks

We propose a novel framework for the regularized inversion of deep neural networks. The framework is based on the authors' recent work on training feed-forward neural networks without the differentiation of activation functions. The framework lifts the parameter space into a higher dimensional space by introducing auxiliary variables, and penalizes these variables with tailored Bregman distances. We propose a family of variational regularizations based on these Bregman distances, present theoretical results and support their practical application with numerical examples. In particular, we present the first convergence result (to the best of our knowledge) for the regularized inversion of a single-layer perceptron that only assumes that the solution of the inverse problem is in the range of the regularization operator, and that shows that the regularized inverse provably converges to the true inverse if measurement errors converge to zero.