Training Feedforward Neural Networks Research Articles

Training feedforward neural networks with Bayesian hyper-heuristics

The process of training feedforward neural networks (FFNNs) can benefit from an automated process where the best heuristic to train the network is sought out automatically by means of a high-level probabilistic-based heuristic. This research introduces a novel population-based Bayesian hyper-heuristic (BHH) that is used to train feedforward neural networks (FFNNs). The performance of the BHH is compared to that of ten popular low-level heuristics, each with different search behaviours. The chosen heuristic pool consists of classic gradient-based heuristics as well as meta-heuristics (MHs). The empirical process is executed on fourteen datasets consisting of classification and regression problems with varying characteristics. The BHH is shown to be able to train FFNNs well and provide an automated method for finding the best heuristic to train the FFNNs at various stages of the training process.

An Enhanced Tunicate Swarm Algorithm with Symmetric Cooperative Swarms for Training Feedforward Neural Networks

The input layer, hidden layer, and output layer are three models of neural processors that comprise feedforward neural networks. In this paper, an enhanced tunicate swarm algorithm based on a differential sequencing alteration operator (ETSA) with symmetric cooperative swarms is presented to train feedforward neural networks. The objective is to accomplish minimum classification errors and the most appropriate neural network layout by regulating the layers’ connection weights and neurons’ deviation thresholds according to the transmission error between the anticipated input and the authentic output. The TSA mimics jet motorization and swarm scavenging to mitigate directional collisions and to maintain the greatest solution that is customized and regional. However, the TSA exhibits the disadvantages of low computational accuracy, a slow convergence speed, and easy search stagnation. The differential sequencing alteration operator has adaptable localized extraction and search screening to broaden the identification scope, enrich population creativity, expedite computation productivity, and avoid search stagnation. The ETSA integrates exploration and exploitation to mitigate search stagnation, which has sufficient stability and flexibility to acquire the finest solution. The ETSA was distinguished from the ETTAO, EPSA, SABO, SAO, EWWPA, YDSE, and TSA by monitoring seventeen alternative datasets. The experimental results confirm that the ETSA maintains profound sustainability and durability to avoid exaggerated convergence, locate the acceptable transmission error, and equalize extraction and prospection to yield a faster convergence speed, superior calculation accuracy, and greater categorization accuracy.

NNBGWO-BRCA marker: Neural Network and binary grey wolf optimization based Breast cancer biomarker discovery framework using multi-omics dataset

Background and ObjectiveBreast cancer is a multifaceted condition characterized by diverse features and a substantial mortality rate, underscoring the imperative for timely detection and intervention. The utilization of multi-omics data has gained significant traction in recent years to identify biomarkers and classify subtypes in breast cancer. This kind of research idea from part to whole will also be an inevitable trend in future life science research. Deep learning can integrate and analyze multi-omics data to predict cancer subtypes, which can further drive targeted therapies. However, there are few articles leveraging the nature of deep learning for feature selection. Therefore, this paper proposes a Neural Network and Binary grey Wolf Optimization based BReast CAncer bioMarker (NNBGWO-BRCAMarker) discovery framework using multi-omics data to obtain a series of biomarkers for precise classification of breast cancer subtypes. MethodsNNBGWO-BRCAMarker consists of two phases: in the first phase, relevant genes are selected using the weights obtained from a trained feedforward neural network; in the second phase, the binary grey wolf optimization algorithm is leveraged to further screen the selected genes, resulting in a set of potential breast cancer biomarkers. ResultsThe SVM classifier with RBF kernel achieved a classification accuracy of 0.9242 ± 0.03 when trained using the 80 biomarkers identified by NNBGWO-BRCAMarker, as evidenced by the experimental results. We conducted a comprehensive gene set analysis, prognostic analysis, and druggability analysis, unveiling 25 druggable genes, 16 enriched pathways strongly linked to specific subtypes of breast cancer, and 8 genes linked to prognostic outcomes. ConclusionsThe proposed framework successfully identified 80 biomarkers from the multi-omics data, enabling accurate classification of breast cancer subtypes. This discovery may offer novel insights for clinicians to pursue in further studies.

Thermo-magnetic radiative flow in porous enclosure with deep-learning parameter estimation

This study numerically investigates convective heat transfer, fluid flow, and entropy generation within an inverted T-shaped porous enclosure filled with a hybrid nanofluid. The interplay between thermal radiation and a magnetic field is considered, as these factors significantly influence thermal acquisition processes in renewable and thermal engineering applications, particularly solar power collectors with the selected physical domain. By analyzing various combinations of flow parameters, this work aims to identify favorable conditions for maximizing heat transfer efficiency in such systems. A Darcy–Forchheimer–Brinkman-based mathematical model has been incorporated and numerically simulated using the penalty finite element method (FEM). Moreover, the variation of fluid flow, isotherms, and entropy generation is thoroughly analyzed for the variation of non-dimensional permeability parameter (Darcy number, Da), buoyancy force (Rayleigh number, Ra), magnetic field (Hartmann number, Ha), porosity value (ϵ), and thermal radiation parameter (Rd). The comprehensive result demonstrates that increasing values of Ra, Da, ϵ, and Rd intensify entropy generation, convective heat, and fluid flow phenomena, whereas the increasing value of Ha substantially weakens these phenomena. The thermal efficiency of the model attends at the lower buoyancy forces (Ra≤105) for each flow parameter (except for Ha). However, the increasing magnetic forces (Ha) substantially improve the thermal efficiency of the developed model. Additionally, this study employs a feed-forward neural network (FNN) based deep learning (DL) approach to predict the parametric influence on convective heat transport rate (Num), total entropy generation rate (Stot), and total Bejan number due to heat transport irreversibility (Beθ,tot) based on the minimally supplied experimental data for FNN training process. The trained FNN model predicts the results very quickly with 98.6% accuracy compared to FEM results, demonstrating its efficiency in approximating the parametric influence of complex mathematical model results. Moreover, this approach significantly reduces computational costs and efforts compared to traditional numerical methods, making it a valuable tool for forecasting the influence of an unknown set of flow parameters in complex mathematical models.

A Hybrid Neural Network‐Based Improved PSO Algorithm for Gas Turbine Emissions Prediction

AbstractIn gas‐fired power plants, emissions may reduce turbine blade rotation, thus decreasing power output. This study proposes a hybrid model integrating the Feed forward Neural Network (FFNN) model and Particle Swarm Optimization (PSO) algorithm to predict gas emissions from natural gas power plants. The FFNN predicts gas turbine nitrogen oxides (NOx) and carbon monoxide (CO) emissions, while the PSO optimizes FFNN weights, improving prediction accuracy. The PSO adopts a unique random number selection strategy, incorporating the K‐Nearest Neighbor (KNN) algorithm to reduce prediction errors. Neighbor Component Analysis (NCA) selects parameters most correlated with CO and NOx emissions. The hybrid model is constructed, trained, and testedusing publicly available datasets, evaluating performance with statistical metrics like Mean Square Error (MSE), Mean Absolute Error (MAE), and Root Mean Square Error (RMSE). Results show significant improvement in FFNN training with the PSO algorithm, boosting CO and NOx prediction accuracy by 99.18% and 82.11%, respectively. The model achieves the lowest MSE, MAE, and RMSE values for CO and NOx emissions. Overall, the hybrid model achieves high prediction accuracy, particularly with optimized PSO parameter selection using seed random generators.

Numerically Optimized Fourier Transform-Based Beamforming Accelerated by Neural Networks

Conventional beamforming methods for reconfigurable reflector antennas assume full control over the amplitude and phase of the reflected field. Here, we develop a novel beamforming methodology for reflecting Programmable Metasurfaces (PMS) with capacitive memory. Although utilizing such fully reactive PMS simplifies antenna design and reduces energy consumption, the PMS reflection magnitude is unity and thus a global optimization of the reflection phases over the PMS unit cells is required in each beamforming scenario. We propose an implementation of such an optimization method rooted in the traditional Fourier transform-based beamforming and evaluate its performance. Additionally, we show that a pair of trained feed-forward neural networks (FFNN) with one input, one hidden, and one output layer can replace time-consuming global optimizations in the case of a PMS comprising 3×10 unit cells. We train the FFNNs on a dataset obtained for typical single- and dual-beam beamforming scenarios. After training, the FFNNs perform requested beamforming tasks within a fraction of second and with about the same accuracy as the original optimization algorithm. The proposed methodology may find applications in future mobile telecommunication systems that require real-time beamforming on low-end hardware. The same beamforming methodology can be also employed in short-range wireless power transfer systems.

A NEURAL NETWORK PREDICTION MODEL FOR SURFACEFINISH IN TURNING PROCESS

This paper presents a neural network based surface finish Prediction model in turning operation .Orthogonal cutting tests were performed on mild steel using H.S.S cutting tool with differentcutting parameters cutting speed ,feed and nose radius of the cutting tool . The collected data wasused to train feed forward back propagation neural network. The developed model has been testedto preclict surface finish for various cutting conditions. The model was found to be powerful &capable of accurate surface finish prediction for the range it had been trained but the accuracydeteriorated as the cutting conditions were changed significantly'

Experimental Results of Underwater Sound Speed Profile Inversion by Few-Shot Multi-Task Learning

Underwater Sound Speed Profile (SSP) distribution is crucial for the propagation mode of acoustic signals, so fast and accurate estimation of SSP is of great importance in building underwater observation systems. The state-of-the-art SSP inversion methods include frameworks of matched field processing (MFP), compressive sensing (CS), and feed-forward neural networks (FNNs), among which the FNN shows better real-time performance while maintaining the same level of accuracy. However, the training of FNN needs quite a lot historical SSP samples, which is difficult to satisfy in many ocean areas. This situation is called few-shot learning. To tackle this issue, we propose a multi-task learning (MTL) model with partial parameter sharing among different training tasks. By MTL, common features could be extracted, which accelerates the learning process on given tasks, and reduces the demand for reference samples, enhancing the generalization ability in few-shot learning. To verify the feasibility and effectiveness of MTL, a deep-ocean experiment was held in April 2023 in the South China Sea. Results show that MTL outperforms the other mainstream methods in terms of accuracy for SSP inversion, while inheriting the real-time advantage of FNN during the inversion stage.

Convergence Analysis and Application for Multi‐Layer Neural Network Based on Fractional‐Order Gradient Descent Learning

AbstractFractional order calculus, with its inheritance and infinite memory properties, is a promising research area in information processing and modeling of certain physical systems, system identification, and control. In this paper, a fractional‐order gradient descent method is proposed for backpropagation training of multilayer feedforward neural networks. In particular, the Caputo derivative is used to define the measurement function and consider the smooth regular term. In addition, the monotonicity of the error function and the strong (weak) convergence theorem of the algorithm are rigorously proved. Finally, numerical experiments prove the correctness and effectiveness of the algorithm theory.

Grey Wolf Cuckoo Search Algorithm for Training Feedforward Neural Network and Logic Gates Design

This paper presents a new hybrid Swarm Intelligence (SI) algorithm based on the Cuckoo Search Algorithm (CSA) and Grey Wolf Optimizer (GWO) called the Grey Wolf Cuckoo Search (GWCS) algorithm. The GWCS algorithm extracts and combines CSA and GWO features for efficient optimization. To carry out the comprehensive validation, the developed algorithm is applied to three different scenarios with their counterparts. The first validation is carried out on standard optimization benchmark problems. Further, they are used to train Feedforward Neural Networks and finally applied to design logic gates. The comprehensive results are presented and it is found that the proposed GWCS algorithms perform better compared to the state-of-the-art.

Comparison of Swarm-based Metaheuristic and Gradient Descent-based Algorithms in Artificial Neural Network Training

This paper aims to compare the gradient descent-based algorithms under classical training model and swarm-based metaheuristic algorithms in feed forward backpropagation artificial neural network training. Batch weight and bias rule, Bayesian regularization, cyclical weight and bias rule and Levenberg-Marquardt algorithms are used as the classical gradient descent-based algorithms. In terms of the swarm-based metaheuristic algorithms, hunger games search, gray wolf optimizer, Archimedes optimization, and the Aquila optimizer are adopted. The Iris data set is used in this paper for the training. Mean square error, mean absolute error and determination coefficient are used as statistical measurement techniques to determine the effect of the network architecture and the adopted training algorithm. The metaheuristic algorithms are shown to have superior capability over the gradient descent-based algorithms in terms of artificial neural network training. In addition to their success in error rates, the classification capabilities of the metaheuristic algorithms are also observed to be in the range of 94%-97%. The hunger games search algorithm is also observed for its specific advantages amongst the metaheuristic algorithms as it maintains good performance in terms of classification ability and other statistical measurements.

Training of Feed-Forward Neural Networks by Using Optimization Algorithms Based on Swarm-Intelligent for Maximum Power Point Tracking.

One of the most used artificial intelligence techniques for maximum power point tracking is artificial neural networks. In order to achieve successful results in maximum power point tracking, the training process of artificial neural networks is important. Metaheuristic algorithms are used extensively in the literature for neural network training. An important group of metaheuristic algorithms is swarm-intelligent-based optimization algorithms. In this study, feed-forward neural network training is carried out for maximum power point tracking by using 13 swarm-intelligent-based optimization algorithms. These algorithms are artificial bee colony, butterfly optimization, cuckoo search, chicken swarm optimization, dragonfly algorithm, firefly algorithm, grasshopper optimization algorithm, krill herd algorithm, particle swarm optimization, salp swarm algorithm, selfish herd optimizer, tunicate swarm algorithm, and tuna swarm optimization. Mean squared error is used as the error metric, and the performances of the algorithms in different network structures are evaluated. Considering the results, a success ranking score is obtained for each algorithm. The three most successful algorithms in both training and testing processes are the firefly algorithm, selfish herd optimizer, and grasshopper optimization algorithm, respectively. The training error values obtained with these algorithms are 4.5 × 10-4, 1.6 × 10-3, and 2.3 × 10-3, respectively. The test error values are 4.6 × 10-4, 1.6 × 10-3, and 2.4 × 10-3, respectively. With these algorithms, effective results have been achieved in a low number of evaluations. In addition to these three algorithms, other algorithms have also achieved mostly acceptable results. This shows that the related algorithms are generally successful ANFIS training algorithms for maximum power point tracking.

On Performance of Marine Predators Algorithm in Training of Feed-Forward Neural Network for Identification of Nonlinear Systems

Artificial neural networks (ANNs) are used to solve many problems, such as modeling, identification, prediction, and classification. The success of ANN is directly related to the training process. Meta-heuristic algorithms are used extensively for ANN training. Within the scope of this study, a feed-forward artificial neural network (FFNN) is trained using the marine predators algorithm (MPA), one of the current meta-heuristic algorithms. Namely, this study is aimed to evaluate the performance of MPA in ANN training in detail. Identification/modeling of nonlinear systems is chosen as the problem. Six nonlinear systems are used in the applications. Some of them are static, and some are dynamic. Mean squared error (MSE) is utilized as the error metric. Effective training and testing results were obtained using MPA. The best mean error values obtained for six nonlinear systems are 2.3 × 10−4, 1.8 × 10−3, 1.0 × 10−4, 1.0 × 10−4, 1.2 × 10−5, and 2.5 × 10−4. The performance of MPA is compared with 16 meta-heuristic algorithms. The results have shown that the performance of MPA is better than other algorithms in ANN training for the identification of nonlinear systems.

Modified Backpropagation Algorithm with Multiplicative Calculus in Neural Networks

Backpropagation is one of the most widely used algorithms for training feedforward deep neural networks. The algorithm requires a differentiable activation function and it performs computations of the gradient proceeding backwards through the feedforward deep neural network from the last layer through to the first layer. In order to calculate the gradient at a specific layer, the gradients of all layers are combined via the chain rule of calculus. One of the biggest disadvantages of the backpropagation is that it requires a large amount of training time. To overcome this issue, this paper proposes a modified backpropagation algorithm with multiplicative calculus. Multiplicative calculus provides an alternative to the classical calculus and it defines new kinds of derivative and integral forms in multiplicative form rather than addition and subtraction forms. The performance analyzes are discussed in various case studies and the results are given comparatively with classical backpropagation algorithm. It is found that the proposed modified backpropagation algorithm converges in less time to the solution and thus provides fast training in the given case studies. It is also shown that the proposed algorithm avoids the local minima problem.

APPSO‐NN: An adaptive‐probability particle swarm optimization neural network for sensorineural hearing loss detection

AbstractAs a hearing disorder, sensorineural hearing loss (SNHL) can be effectively detected by magnetic resonance imaging (MRI). However, the manual detection of MRI scanning is subjective, time‐consuming, and unpredictable. An accurate and automatic computer‐aided diagnosis system is proposed for SNHL detection, providing reliable references for professionals. The system first employs a wavelet entropy layer to extract features of MRI images. Then, a neural network layer is proposed as the classifier consisting of a feedforward neural network (FNN) and an adaptive‐probability PSO (APPSO) algorithm. The authors prove the rotation‐variant property of the basic particle swarm optimization (PSO) by the algebraic property of matrix transformation. The property is unsuitable for optimising parameters of neural networks. Thus, in APPSO, the authors integrate the new update rules based on all‐dimensional variation and adaptive‐probability mechanism into the basic PSO, which can improve its searching ability without losing population diversity. The authors compare APPSO‐NN with FNN trained by five popular evolutionary algorithms. The simulation results show that APPSO performs best in training FNN. The method also compares with six state‐of‐the‐art methods. The simulation results show that the best performance in sensitivity and overall accuracy of hearing loss classification, which proves that the method is effective and promising for SNHL detection.

A lifted Bregman formulation for the inversion of deep neural networks

We propose a novel framework for the regularized inversion of deep neural networks. The framework is based on the authors' recent work on training feed-forward neural networks without the differentiation of activation functions. The framework lifts the parameter space into a higher dimensional space by introducing auxiliary variables, and penalizes these variables with tailored Bregman distances. We propose a family of variational regularizations based on these Bregman distances, present theoretical results and support their practical application with numerical examples. In particular, we present the first convergence result (to the best of our knowledge) for the regularized inversion of a single-layer perceptron that only assumes that the solution of the inverse problem is in the range of the regularization operator, and that shows that the regularized inverse provably converges to the true inverse if measurement errors converge to zero.

On Correspondences between Feedforward Artificial Neural Networks on Finite Memory Automata and Classes of Primitive Recursive Functions

When realized on computational devices with finite quantities of memory, feedforward artificial neural networks and the functions they compute cease being abstract mathematical objects and turn into executable programs generating concrete computations. To differentiate between feedforward artificial neural networks and their functions as abstract mathematical objects and the realizations of these networks and functions on finite memory devices, we introduce the categories of general and actual computabilities and show that there exist correspondences, i.e., bijections, between functions computable by trained feedforward artificial neural networks on finite memory automata and classes of primitive recursive functions.

Integrating modeled environmental variability into neural network training for underwater source localization.

Supervised machine learning (ML) is a powerful tool that has been applied to many fields of underwater acoustics, including acoustic inversion. ML algorithms depend on the existence of extensive labeled datasets, which are difficult to obtain for the task of underwater source localization. A feed-forward neural network (FNN) trained on imbalanced or biased data may end up suffering from a problem analogous to model mismatch in matched field processing (MFP), that is, producing incorrect results due to a difference between the environment sampled by the training data and the actual environment. To overcome this issue, physical and numerical propagation models can act as data augmentation tools to compensate for the lack of comprehensive acoustic data. This paper examines how modeled data can be effectively used for training FNNs. Mismatch tests compare the output from a FNN and MFP and show that the network becomes more robust to various kinds of mismatches when trained on diverse environments. A systematic analysis of how the training dataset's variability impacts a FNN's localization performance on experimental data is carried out. Results show that networks trained with synthetic data achieve better and more robust performance than regular MFP when environment variability is taken into account.

Quantized rewiring: hardware-aware training of sparse deep neural networks

Mixed-signal and fully digital neuromorphic systems have been of significant interest for deploying spiking neural networks in an energy-efficient manner. However, many of these systems impose constraints in terms of fan-in, memory, or synaptic weight precision that have to be considered during network design and training. In this paper, we present quantized rewiring (Q-rewiring), an algorithm that can train both spiking and non-spiking neural networks while meeting hardware constraints during the entire training process. To demonstrate our approach, we train both feedforward and recurrent neural networks with a combined fan-in/weight precision limit, a constraint that is, for example, present in the DYNAP-SE mixed-signal analog/digital neuromorphic processor. Q-rewiring simultaneously performs quantization and rewiring of synapses and synaptic weights through gradient descent updates and projecting the trainable parameters to a constraint-compliant region. Using our algorithm, we find trade-offs between the number of incoming connections to neurons and network performance for a number of common benchmark datasets.

Testing Feedforward Neural Networks Training Programs

At present, we are witnessing an increasing effort to improve the performance and trustworthiness of Deep Neural Networks (DNNs), with the aim to enable their adoption in safety critical systems such as self-driving cars or aircraft collision-avoidance systems. Multiple testing techniques are proposed to generate test cases that can expose inconsistencies in the behavior of DNN models. These techniques assume implicitly that the training program is bug-free and appropriately configured. However, satisfying this assumption for a novel problem requires significant engineering work to prepare the data, design the DNN, implement the training program, and tune the hyperparameters to produce the model for which current automated test data generators search for corner-case behaviors. All these model training steps can be error prone. Therefore, it is crucial to detect and correct errors throughout all the engineering steps of DNN-based software systems and not only on the resulting DNN model. In this article, we gather a catalog of training issues and based on their symptoms and their effects on the behavior of the training program, we propose practical verification routines to detect the aforementioned issues, automatically, by continuously validating that some important properties of the learning dynamics hold during the training. Then, we design TheDeepChecker , an end-to-end property-based debugging approach for DNN training programs and implement it as a TensorFlow-based library. As an empirical evaluation, we conduct a case study to assess the effectiveness of TheDeepChecker on synthetic and real-world buggy DL programs and compare its performance to that of the Amazon SageMaker Debugger ( SMD ). Results show that TheDeepChecker ’s on-execution validation of DNN-based program’s properties through three sequential phases (pre-, on-, and post-fitting) succeeds in revealing several coding bugs and system misconfigurations errors early on and at a low cost. Moreover, our property-based approach outperforms the SMD ’s offline rules verification on training logs in terms of detection accuracy for unstable learning issues and coverage of additional DL bugs.