Backpropagation Learning Algorithm Research Articles

Comparative study of integer-order and fractional-order artificial neural networks: Application for mathematical function generation

This paper investigates the impact of fractional derivatives on the activation functions of an artificial neural network (ANN). Based on the results and analysis, a three-layer backpropagation neural network model with fractional and integer derivatives employed in the activation function and fractional gradient descent method backpropagation learning algorithm has been proposed. Specifically, three perceptrons have been proposed based on fractional and integer derivatives applied to activation functions and learning algorithms. They are fractional derivative activation function (FDAF) perceptron, integer derivative activation function (IDAF)-fractional derivative learning algorithm (FDLA) perceptron, and fractional derivative learning algorithm (FDLA) perceptron. The Riemann-Liouville (RL) fractional derivative, Grunwald-Letnikov (GL) derivative, Caputo-Fabrizio (CF) fractional derivative, Caputo (C) fractional derivative, and Atangana-Baleanu (AB) fractional derivative have been employed. The impact of these derivatives’ fractional-order (FO) is investigated in a wide range from 0.1−0.9 on testing mean square error (MSE) and time required to train the perceptron. FO-based and IO-based perceptrons are compared with the help of performance metrics such as testing MSE and the time required to train the models. The training and testing simulation results illustrate that Caputo-Fabrizio derivative-based perceptrons outperform other fractional derivatives. Also, the performance of the FO-based perceptron is better in terms of the least MSE.

Prediction of pressure evolution in non-venting self-pressurized liquid hydrogen tanks using artificial neural network approach

Liquid hydrogen (LH2) is a future energy carrier compared to most conventional fuels. One of the practical bottlenecks in non-venting LH2 tanks is to accurately predict the pressure evolution. Accordingly, many thermal models and correlations were developed; however significant deviations arose due to inherent complex thermal phenomena which have not been addressed yet. A general artificial neural network (ANN) model has been first developed based on 448 datasets amassed from different literature sources to predict the pressure evolution under different heat fluxes, initial filling ratios, as well as tank volumes and shapes. The model has been configured with a typical multi-layer feed-forward structure using a back-propagation learning algorithm and Levenberg-Marquardt training algorithm. For the ANN model, a preliminary optimization process has been conducted for reaching the optimal values for training-to-testing datasets ratio, learning parameters and optimum network structure. Besides, some criteria were set to evaluate the ANN model performance. The ANN model with the configuration of 5-11-11-1 was found to be adopted for such a prediction task. Compared to experimental results and literature models and correlations, the ANN model has been proven to predict the pressure evolution in non-venting self-pressurized LH2 tanks with a mean square error of 1.1779 × 10−5 s2, root mean square error of 0.0034 s, mean absolute error of 6%, mean relative error of 0.2% as well as a satisfactory correlation coefficient of 0.99934 between the network outputs and corresponding targets. This research work provides an effective modeling approach based on artificial neural network to solve one complex problem related with LH2 tanks with fast, accurate and consistent results. Also, the results of the present work can offer some practical guidance for the design of LH2 tanks. Further, the present work gives some insights into new application areas of ANN which will be the emphasis of future research work.

Artificial Neural Network for Automated Keratoconus Detection Using a Combined Placido Disc and Anterior Segment Optical Coherence Tomography Topographer.

To assess the efficacy of an automated program for keratoconus and keratoconus suspect detection based on corneal measurements provided by a combined Placido disc and anterior segment optical coherence tomography (OCT) topographer. In a multicentric cross-sectional study, an artificial neural network (ANN) was created using 6677 eyes from an equal number of patients (classified as 2663 normal eyes, 1616 keratoconus eyes, 210 keratoconus suspect eyes, 1519 myopic postoperative eyes, and 669 abnormal eyes). Each group was randomly divided into a training set (70% of the dataset) and a validation set (the remaining 30%). A multilayer perceptron network with a backpropagation learning algorithm was developed for the study. Indexes used to train the ANN were based on curvature and elevation of both the anterior and posterior corneal surfaces and the new corneal OCT indexes-based on corneal, stromal, and epithelial thicknesses. For keratoconus detection, our ANN showed an accuracy of 98.6%, precision of 96%, recall of 97.9%, and F1-score of 96.9%. For keratoconus suspect detection, our ANN showed an accuracy of 98.5%, precision of 83.6%, recall of 69.7%, and F1-score of 76%. Compared to previous literature, the addition of new OCT-based epithelial and stromal thickness indexes improves ANN detection capacity of keratoconus suspect eyes. For already stablished keratoconus our ANN detection capacity is excellent, but equivalent to previous evidence without incorporating such new OCT-based indexes. OCT-based epithelial and stromal thickness indexes improve ANN detection capacity of keratoconus on its early stages.

Design of backpropagation learning algorithm for MHD mixed convective Prandtl nanofluid flow with activation energy

Design of backpropagation learning algorithm for MHD mixed convective Prandtl nanofluid flow with activation energy

An applied deep reinforcement learning approach to control active networked microgrids in smart cities with multi-level participation of battery energy storage system and electric vehicles

This study proposed an intelligent energy management strategy for islanded networked microgrids (NMGs) in smart cities considering the renewable energy sources uncertainties and power fluctuations. Energy management of active power and frequency control approach is based on the intelligent probabilistic wavelet fuzzy neural network-deep reinforcement learning algorithm (IPWFNN-DRLA). The control strategy is formulated with deep reinforcement learning approach based on the Markov decision process and solved by the soft actor-critic algorithm. The NMG local controller (NMGLC) provides information such as the frequency, active power, power generation data, and status of the electric vehicle's battery energy storage system to the NMG central controller (NMGCC). Then the NMGCC calculates active power and frequency support based on the IPWFNN-DRLA approach and sends the results to the NMGLC. The proposed model is developed in a continuous problem-solving space with two structures of offline training and decentralized distributed operation. For this purpose, each NMG has a control agent (NMGCA) based on the IPWFNN algorithm, and the NMGCA learning model is formulated based the online back-propagation learning algorithm. The proposed approach demonstrates a computation accuracy exceeding 98%, along with a 7.82% reduction in computational burden and a 61.1% reduction in computation time compared to alternative methods.

Mathematical Formulation of Learning and Its Computational Complexity for Transformers’ Layers

Transformers are the cornerstone of natural language processing and other much more complicated sequential modelling tasks. The training of these models, however, requires an enormous number of computations, with substantial economic and environmental impacts. An accurate estimation of the computational complexity of training would allow us to be aware in advance about the associated latency and energy consumption. Furthermore, with the advent of forward learning workloads, an estimation of the computational complexity of such neural network topologies is required in order to reliably compare backpropagation with these advanced learning procedures. This work describes a mathematical approach, independent from the deployment on a specific target, for estimating the complexity of training a transformer model. Hence, the equations used during backpropagation and forward learning algorithms are derived for each layer and their complexity is expressed in the form of MACCs and FLOPs. By adding all of these together accordingly to their embodiment into a complete topology and the learning rule taken into account, the total complexity of the desired transformer workload can be estimated.

Exploring the relationship between space weather parameters and cosmic ray muons observed at high cut-off rigidity site: A correlation, artificial neural network, and spectral analysis

The objective of this study was to examine the relationship between cosmic rays (CR) and various solar activity parameters, including the interplanetary magnetic field (B), solar radio emission flux at 10.7 cm (F10.7), solar wind speed (V), solar wind density (N), solar wind pressure (P), and the variance of the interplanetary magnetic field vector (σF). The CR data collected from 2007 to 2013 were obtained using a muon detector situated at King Abdulaziz University (KAAU) in Jeddah, Saudi Arabia, with a rigidity cutoff of 14.8 GV. To explore the correlation between the CR data and the selected parameters, three distinct methods were employed: regression analysis, artificial neural network (ANN), and power spectral analysis.Regression analysis revealed significant positive correlations between CR muons and plasma pressure (R = 0.1, p < 0.05), as well as solar wind speed (R = 0.1, p < 0.05). On the other hand, negative correlations were observed between CRs and radio flux (R = 0.75, p < 0.05), as well as interplanetary magnetic field (R = 0.1, p < 0.05). However, no significant correlations were found between CRs and plasma density or the variance of the interplanetary magnetic field vector (p > 0.05). Furthermore, multivariable correlation analyses were conducted to explore the relationship between CR muons and the considered parameters, resulting in only marginal improvement.For comparison purposes, data from the King Abdulaziz City for Science and Technology (KACST) muon detector in Riyadh, Saudi Arabia (with a rigidity cutoff of 14.4 GV), as well as CR data from Oulu NM (with a rigidity cutoff of 0.3 GV), were utilized for the same period as the KAAU data. The correlation results indicated that the relationships between the CR data from the three detectors and some of the considered parameters were consistent. However, the strength of the correlations and the magnitudes of the slopes exhibited variations.ANN procedures were also employed to compare the efficiency of neural network analysis in forecasting CR muons based on solar activity parameters. The Multilayer Perceptron (MLP) Module trained using the backpropagation learning algorithm was utilized to build a neural network model and assess its accuracy. The results indicated that the ANN method provided slightly more accurate predictions compared to regression analysis.Finally, Fast Fourier Transform (FFT) analyses were conducted to examine the presence of periodicities between the CRs and the variables under consideration. The analysis revealed several shared periodicities, including cycles spanning 270–277 days, 146–158 days, 89–95 days, 66–73 days, and 21–31 days.

Research of the numerical simulation and machine learning backpropagation networks analysis of the sound absorption properties of cellular soundproofing materials

Cellular soundproofing materials have been critical for lowering incidence pressure waves, which have led to their wide application in studios, auditoria, automotive and aerospace applications, among others. This work, therefore, premiers the utilisation of machine learning backpropagation algorithm of artificial neural networks (ANN) to develop and train a series of datasets of sound absorption properties (noise reduction coefficient, sound absorption average, and area under the sound absorption spectrum) obtained via numerical modelling and simulations. ANN models were established to analyze these sound absorption properties in relation to several non–acoustical parameters associated with cellular materials, including their thickness. Modelling and simulation revealed the relative importance of non–acoustic parameters on sound absorption spectra for these structures. ANN models predicted output signals that almost overlapped with their respective “real” signals with correlation coefficients over 95 %. Model confidence was proposed to analytically predict sound absorption phenomena accurately for cellular materials characterized by permeability ranging between 0.5–3.0 × 10−09 m2 or viscous characteristic lengths ranging between 80 and 300 μm. Analytical models based on optimised synaptic weights, biases, non–acoustic parameters, and material thickness could assist manufacturers design soundproofing materials for noise reduction and reverberation control.

Binary‐Stochasticity‐Enabled Highly Efficient Neuromorphic Deep Learning Achieves Better‐than‐Software Accuracy

In this work, the requirement of using high‐precision (HP) signals is lifted and the circuits for implementing deep learning algorithms in memristor‐based hardware are simplified. The use of HP signals is required by the backpropagation learning algorithm since the gradient descent learning rule relies on the chain product of partial derivatives. However, it is both challenging and biologically implausible to implement such an HP algorithm in noisy and analog memristor‐based hardware systems. Herein, it is demonstrated that the requirement for HP signals handling is not necessary and more efficient deep learning can be achieved when using a binary stochastic learning algorithm. The new algorithm proposed in this work modifies elementary neural network operations, which improves energy efficiency by two orders of magnitude compared to traditional memristor‐based hardware and three orders of magnitude compared to complementary metal–oxide–semiconductor‐based hardware. It also provides better accuracy in pattern recognition tasks than the HP learning algorithm benchmarks.

Deploying artificial neural network to predict hybrid biodiesel fuel properties from their fatty acid compositions

Measurement-related problems have spurred fuel properties prediction using machine learning techniques. Improved fuel properties offered by hybrid biodiesel (HB) via mixed oils were predicted from their fatty acid compositions (FACs) using artificial neural network (ANN). FACs and fuel properties of HB sourced from the literature were used to develop ANN models. FAC data were used as the input parameters to predict the fuel properties data (kinematic viscosity (KV), density, calorific value (CV), and flash point (FP)) considered as the output parameters of the models. Using the multilayer perception ANN, the models were trained using Levenberg-Marquardt back propagation learning algorithm coupled with different numbers of neurons and activation functions for the prediction of the fuel properties. The models were observed to accurately predict these fuel properties with high prediction accuracy (R 2 = 1). The evaluated model performance errors were 0.1014 and 0.0504, 0.2905 and 0.4225, 0.1848, and 0.1038, and 0.4726 and 0.7833 for KV, density, CV, and FP using root mean square error and average absolute deviation respectively. Prediction performance and error estimates were slightly better than those for single feedstock biodiesel. Hence, this study shows the ability of ANN to accurately predict the fuel properties of HB from the FAs.

Understanding neural network through neuron level visualization

Neurons are the fundamental units of neural networks. In this paper, we propose a method for explaining neural networks by visualizing the learning process of neurons. For a trained neural network, the proposed method obtains the features learned by each neuron and displays the features in a human-understandable form. The features learned by different neurons are combined to analyze the working mechanism of different neural network models. The method is applicable to neural networks without requiring any changes to the architectures of the models. In this study, we apply the proposed method to both Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs) trained using the backpropagation learning algorithm. We conduct experiments on models for image classification tasks to demonstrate the effectiveness of the method. Through these experiments, we gain insights into the working mechanisms of various neural network architectures and evaluate neural network interpretability from diverse perspectives.

Internal force minimization-based adaptive beam string structure: Numerical analysis and experimental study

In this study, a novel adaptive beam string structure (ABSS) designed to cope with external loading through sensing and mechanical actuation was developed. Unlike traditional beam string structures, ABSS further contains the monitoring and controlling systems, which allows it to sense and react to loading in real-time. The active control method for ABSS based on internal force minimization was established by using the strategy search algorithm (SSA) and strategy prediction network (SPN). SSA is a global search algorithm proposed through combining the strengths of genetic algorithm and gradient descent, and SPN is an artificial neural network established by applying the backpropagation learning algorithm. The downscaled model and test scheme of ABSS were designed and deployed to clarify its effectiveness. The numerical simulation and field experiments were conducted, and simulation results agree well with measurements. Through comparative analysis of simulation and test results, the adaption of ABSS to loading and validity of control method were verified, and ABSS possesses good control capability on internal forces, which is reflected in two ways: 1) reducing the beam stress responses; and 2) improving the beam stress distribution. The research methods and conclusions of this paper can provide valuable references for the design, fabrication, and control of adaptive structures.

Modeling capacitance of carbon-based supercapacitors by artificial neural networks

Carbon-based electrodes effectively promote the specific capacitance of the supercapacitors. The Specific capacitance of carbon-based electrodes has been modeled using an artificial neural network (ANN) with the backpropagation learning algorithm. This paper describes the creation of an ANN model to interpret how voltage window (V), ID/IG, N/O-dopings (at. %), pore size (nm), and specific surface area (m2/g) parameters influence the specific capacitance (F/g). The experimentation has been carried out with several ANN architectures to achieve the best fit between the inputs and output. The model predictions (adj.R2 = 0.99) and estimation of the isolated effect of independent variables, such as voltage window, cannot be varied independently in practice. The results from the ANN model were consistent with the existing theory and reasonable in estimating the specific capacitance beyond the scope of the experimental data. The model successfully expresses the specific capacitance of carbon-based supercapacitors as a function of physiochemical and electrochemical process variables and can be used to design electrical storage devices.

Artificial neural network based load balancing scheme for top of rack switches in optical data centers

Abstract Data centers serve as dedicated facilities for housing computer systems and their related components, including telecommunications and storage systems. They typically have high levels of security and environmental controls to ensure that the equipment housed within them functions optimally. Data center networks (DCNs) often employ load balancing algorithms to handle large volumes of traffic and ensure that all servers and switches are utilized equally, keeping the network running smoothly. However, as load on the server varies, therefore dynamic traffic management systems that can adjust traffic flow in real-time based on the current traffic state is required. This study presents an artificial neural network-based load balancing method. By training a feed-forward artificial neural network (ANN) using a back propagation (BP) learning algorithm, it evenly distributes workload over all of the nodes. Simulation results are also presented to prove the usefulness of the proposed load balancing mechanism. It is found that the load balancing scheme can reduce the packet blocking probability (PBP) by 10 folds and delay by about nearly 11 percent.

Design and Simulation of a Multilayer Chemical Neural Network That Learns via Backpropagation.

The design and implementation of adaptive chemical reaction networks, capable of adjusting their behavior over time in response to experience, is a key goal for the fields of molecular computing and DNA nanotechnology. Mainstream machine learning research offers powerful tools for implementing learning behavior that could one day be realized in a wet chemistry system. Here we develop an abstract chemical reaction network model that implements the backpropagation learning algorithm for a feedforward neural network whose nodes employ the nonlinear "leaky rectified linear unit" transfer function. Our network directly implements the mathematics behind this well-studied learning algorithm, and we demonstrate its capabilities by training the system to learn a linearly inseparable decision surface, specifically, the XOR logic function. We show that this simulation quantitatively follows the definition of the underlying algorithm. To implement this system, we also report ProBioSim, a simulator that enables arbitrary training protocols for simulated chemical reaction networks to be straightforwardly defined using constructs from the host programming language. This work thus provides new insight into the capabilities of learning chemical reaction networks and also develops new computational tools to simulate their behavior, which could be applied in the design and implementations of adaptive artificial life.

A hybrid approach for forecasting bitcoin series

Bitcoin price prediction is a substantial challenge for cryptocurrency investors. This study offers an innovative scheme to predict Bitcoin returns and volatilities using a hybrid model that incorporates the autoregressive fractionally integrated moving average (ARFIMA), empirical wavelet (EW) transform, and local linear wavelet neural network (LLWNN) approaches to produce an ARFIMA-EWLLWNN model. Our methodologies integrate the advantages of the long-memory model, EW decomposition technique, artificial neural network structure, and backpropagation and particle swarm optimization learning algorithms. The experimental results of the optimized hybrid approach outperform some classic models by providing more accurate out-of-sample forecasts over longer horizons. The model proves to be the most appropriate Bitcoin forecasting technique. Moreover, the implemented method produces smaller prediction errors than other computing techniques.

Fault Location of Doukan-Erbil 132kv Double Transmission Lines Using Artificial Neural Network ANN

Transmission lines are generally subjected to faults, so it is advantageous to determine these faults as quickly as possible. This study uses an Artificial Neural Network technique to locate a fault as soon as it happens on the Doukan-Erbil of 132kv double Transmission lines network. CYME 7.1-Programming/Simulink utilized simulation to model the suggested network. A multilayer perceptron feed-forward artificial neural network with a back propagation learning algorithm is used for the intelligence locator's training, testing, assessment, and validation. Voltages and currents were applied as inputs during the neural network's training. The pre-fault and post-fault values determined the scaled values. The neural network's performance was evaluated, and tests were run. Line-to-ground faults were examined. The study demonstrates how effective, rapid, and precise this method is at locating faults. The neural network's performance was examined, and tests were run on it. The overall performance of the mean square error in the trained network execution was 0.11792 at 35 epochs. The correlation coefficient at the entire target was 0.99987 percent of an error on the Doukan-Erbil double transmission lines.

Intelligent Arabic letters speech recognition system based on mel frequency cepstral coefficients

&lt;span lang="EN-US"&gt;Speech recognition is one of the important applications of artificial intelligence (AI). Speech recognition aims to recognize spoken words regardless of who is speaking to them. The process of voice recognition involves extracting meaningful features from spoken words and then classifying these features into their classes. This paper presents a neural network classification system for Arabic letters. The paper will study the effect of changing the multi-layer perceptron (MLP) artificial neural network (ANN) properties to obtain an optimized performance. The proposed system consists of two main stages; first, the recorded spoken letters are transformed from the time domain into the frequency domain using fast Fourier transform (FFT), and features are extracted using mel frequency cepstral coefficients (MFCC). Second, the extracted features are then classified using the MLP ANN with back-propagation (BP) learning algorithm. The obtained results show that the proposed system along with the extracted features can classify Arabic spoken letters using two neural network hidden layers with an accuracy of around 86%.&lt;/span&gt;

A publicly available PyTorch-ABAQUS UMAT deep-learning framework for level-set plasticity

This paper introduces a publicly available PyTorch-ABAQUS deep-learning framework of a family of plasticity models where the yield surface is implicitly represented by a scalar-valued function. In particular, our focus is to introduce a practical framework that can be deployed for engineering analysis that employs a user-defined material subroutine (UMAT/VUMAT) for ABAQUS, which is written in FORTRAN. To accomplish this task while leveraging the back-propagation learning algorithm to speed up the neural-network training, we introduce an interface code where the weights and biases of the trained neural networks obtained via the PyTorch library can be automatically converted into a generic FORTRAN code that can be a part of the UMAT/VUMAT algorithm. To enable third-party validation, we purposely make all the data sets, source code used to train the neural-network-based constitutive models, and the trained models available in a public repository. Furthermore, the practicality of the workflow is then further tested on a dataset for anisotropic yield function to showcase the extensibility of the proposed framework. A number of representative numerical experiments are used to examine the accuracy, robustness and reproducibility of the results generated by the neural network models.

An Analysis of the Influence of Surface Roughness and Clearance on the Dynamic Behavior of Deep Groove Ball Bearings Using Artificial Neural Networks.

The deep groove ball bearing is one of the most important components of the rotary motion system and is the research subject in this paper. After factory assembly, new ball bearings need to pass quality control. The conventional approach relies on measuring the vibration amplitudes for each unit and sorting them into classes according to the vibration level. In this paper, based on experimental research, models are created to predict the vibration class and analyze the dynamic behavior of new ball bearings. The models are based on artificial neural networks. A feedforward multilayer perceptron (MLP) was applied, and a backpropagation learning algorithm was used. A specific method of training groups of artificial neural networks was applied, where each network provided an answer to the input within the group, and the final answer was the mean value of the answers of all networks in the group. The models achieved a prediction accuracy of over 90%. The main aim of the research was to construct models that are able to predict the vibration class of a new ball bearing based on the geometric parameters of the bearing rings. The models are also applied to analyze the influence of surface roughness of the raceways and the internal radial clearance on bearing vibrations.