Backpropagation Algorithm Research Articles

The Symplectic Adjoint Method: Memory-Efficient Backpropagation of Neural-Network-Based Differential Equations.

The combination of neural networks and numerical integration can provide highly accurate models of continuous-time dynamical systems and probabilistic distributions. However, if a neural network is used n times during numerical integration, the whole computation graph can be considered as a network n times deeper than the original. The backpropagation algorithm consumes memory in proportion to the number of uses times of the network size, causing practical difficulties. This is true even if a checkpointing scheme divides the computation graph into subgraphs. Alternatively, the adjoint method obtains a gradient by a numerical integration backward in time; although this method consumes memory only for single-network use, the computational cost of suppressing numerical errors is high. The symplectic adjoint method proposed in this study, an adjoint method solved by a symplectic integrator, obtains the exact gradient (up to rounding error) with memory proportional to the number of uses plus the network size. The theoretical analysis shows that it consumes much less memory than the naive backpropagation algorithm and checkpointing schemes. The experiments verify the theory, and they also demonstrate that the symplectic adjoint method is faster than the adjoint method and is more robust to rounding errors.

Automatic Learning Rate Adaption for Memristive Deep Learning Systems.

As a possible device to further enhance the performance of the hybrid complementary metal oxide semiconductor (CMOS) technology in the hardware, the memristor has attracted widespread attention in implementing efficient and compact deep learning (DL) systems. In this study, an automatic learning rate tuning method for memristive DL systems is presented. Memristive devices are utilized to adjust the adaptive learning rate in deep neural networks (DNNs). The speed of the learning rate adaptation process is fast at first and then becomes slow, which consist of the memristance or conductance adjustment process of the memristors. As a result, no manual tuning of learning rates is required in the adaptive back propagation (BP) algorithm. While cycle-to-cycle and device-to-device variations could be a significant issue in memristive DL systems, the proposed method appears robust to noisy gradients, various architectures, and different datasets. Moreover, fuzzy control methods for adaptive learning are presented for pattern recognition, such that the over-fitting issue can be well addressed. To our best knowledge, this is the first memristive DL system using an adaptive learning rate for image recognition. Another highlight of the presented memristive adaptive DL system is that quantized neural network architecture is utilized, and there is therefore a significant increase in the training efficiency, without the loss of testing accuracy.

Efficient design of complex-valued neural networks with application to the classification of transient acoustic signals.

A paper by the current authors Paul and Nelson [JASA Express Lett. 3(9), 094802 (2023)] showed how the singular value decomposition (SVD) of the matrix of real weights in a neural network could be used to prune the network during training. The paper presented here shows that a similar approach can be used to reduce the training time and increase the implementation efficiency of complex-valued neural networks. Such networks have potential advantages compared to their real-valued counterparts, especially when the complex representation of the data is important, which is the often case in acoustic signal processing. In comparing the performance of networks having both real and complex elements, it is demonstrated that there are some advantages to the use of complex networks in the cases considered. The paper includes a derivation of the backpropagation algorithm, in matrix form, for training a complex-valued multilayer perceptron with an arbitrary number of layers. The matrix-based analysis enables the application of the SVD to the complex weight matrices in the network. The SVD-based pruning technique is applied to the problem of the classification of transient acoustic signals. It is shown how training times can be reduced, and implementation efficiency increased, while ensuring that such signals can be classified with remarkable accuracy.

Advancing Reservoir Evaluation: Machine Learning Approaches for Predicting Porosity Curves

Porosity assessment is a vital component for reservoir evaluation in the oil and gas sector, and with technological advancement, reliance on conventional methods has decreased. In this regard, this research aims to reduce reliance on well logging, purposing successive machine learning (ML) techniques for precise porosity measurement. So, this research examines the prediction of the porosity curves in the Sui main and Sui upper limestone reservoir, utilizing ML approaches such as an artificial neural networks (ANN) and fuzzy logic (FL). Thus, the input dataset of this research includes gamma ray (GR), neutron porosity (NPHI), density (RHOB), and sonic (DT) logs amongst five drilled wells located in the Qadirpur gas field. The ANN model was trained using the backpropagation algorithm. For the FL model, ten bins were utilized, and Gaussian-shaped membership functions were chosen for ideal correspondence with the geophysical log dataset. The closeness of fit (C-fit) values for the ANN ranged from 91% to 98%, while the FL model exhibited variability from 90% to 95% throughout the wells. In addition, a similar dataset was used to evaluate multiple linear regression (MLR) for comparative analysis. The ANN and FL models achieved robust performance as compared to MLR, with R2 values of 0.955 (FL) and 0.988 (ANN) compared to 0.94 (MLR). The outcomes indicate that FL and ANN exceed MLR in predicting the porosity curve. Moreover, the significant R2 values and lowest root mean square error (RMSE) values support the potency of these advanced approaches. This research emphasizes the authenticity of FL and ANN in predicting the porosity curve. Thus, these techniques not only enhance natural resource exploitation within the region but also hold broader potential for worldwide applications in reservoir assessment.

Classification of Crude Palm Oil Quality Using Artificial Neural Networks Based on Chemical Components

Palm oil, known as Crude Palm Oil (CPO), is a flagship product in the palm oil industry, playing a crucial role in the economies of various tropical countries such as Indonesia. This study aims to classify the quality of CPO based on chemical components using the Artificial Neural Networks (ANN) method with a backpropagation algorithm. The research data was obtained from PT. Perkebunan Lembah Bhakti (PLB) 2 from January to October 2023, consisting of 225 entries with five main chemical variables: impurity level, moisture content, free fatty acid (FFA) level, Deterioration of Bleachability Index (DOBI), and carotenoids. The data preprocessing stage involved transforming and normalizing the data using the Z-score method. The ANN model used has a 5-5-2 architecture with ReLU activation functions for the input and hidden layers and a Softmax function for the output layer. Model evaluation was conducted using accuracy metrics, which showed that the ANN model could classify CPO quality with an accuracy of 97.78% on the test data. The research results show that ANN can classify CPO quality with a high level of accuracy. This indicates that this method has great potential for use in the industry to improve CPO quality assessment. The improvement in accuracy and validity of the ANN classification results has significant implications for the industry. With high accuracy, ANN can reduce human errors in quality assessment, speed up the process, and increase the consistency of assessment results. This is crucial because consistent quality assessment can enhance operational efficiency and reduce production costs.

Implementation of Artificial Neural Network in Predicting CPO Prices Using Backpropagation

This research examines the development of a forecasting model for Crude Palm Oil (CPO) prices using artificial neural network algorithms, particularly the backpropagation algorithm. CPO, as Indonesia's main export commodity, has significant economic impacts and affects the income of oil palm farmers. Data on CPO prices taken from CIF Rotterdam from January 2019 to December 2023 were used in this study. The research method involved several stages, including data collection, pre-processing, model design, and model implementation using Python programming. The results of training the model using the backpropagation algorithm showed an error value of 0.537829578 after 1000 epochs, while evaluation using Mean Squared Error (MSE) showed an MSE value of 0.022709 during the training process and 0.017604 during the testing process. The model also produced predictions for CPO prices in the next three months: 932.578 for the first month, 949.568 for the second month, and 774.855 for the third month. These findings indicate that the developed model can predict future CPO prices with adequate accuracy, which can assist companies in making better financial decisions and managing risks associated with CPO price fluctuations.

Design of stochastic computational Levenberg Marquardt backpropagation-based technique to investigate temperature distribution of longitudinal moving porous fin

The improvement of thermal exchange is of utmost interest in a wide range of engineering areas. The current study focuses on thermal evaluation involving natural radiation and convection in a fractionally arranged moving longitudinal fin model placed under a magnetic field. We implement the Levenberg Marquardt backpropagation (LMB) algorithm for investigating an innovative use of stochastic numerical computation for analyzing the efficiency of the temperature distribution in a porous moving longitudinal fin. The datasets for LMB have been created using a shooting approach for dynamic systems with varying ranges of different parameters. The validation, testing, and training processes are used to simulate networks using the LMB approach for diverse scenarios of moving porous fin models. The reliability of results is assessed based on the regression measures, absolute error, error histograms, mean square error, and other metrics for fuller numerical modeling of the suggested LMB to investigate the thermal efficiency and effectiveness of porous moving fin.

Predicting green tea moisture content during roasting by shape feature

This study investigated the changes in tea moisture content and shape features parameters during the roasting process of roasted green tea under different temperatures(70, 100 and 130 °C), leaf amounts (40, 70 and 100 kg), and cylinder speeds (25, 27.5 and 30 r/min), and developed a moisture content prediction model based on Random forest-Genetic algorithm-Back propagation(RF-GA-BP) neural network. Firstly, based on image processing technology, the shape parameters of tea leaves during the roasting process were automatically extracted, and the random forest algorithm was used to rank their importance to reduce the complexity of the model. Then, the activation function and the number of hidden layer nodes in the network structure were optimized. Finally, the initial weights and thresholds of the neural network were optimized using genetic algorithm to obtain the optimal parameters for constructing a moisture content prediction model during the roasting process. The results indicated that the importance of slightness was the highest, with a value of 0.865. The RF-GA-BP neural network model with mean absolute error, root mean square error, and coefficient of determination of 0.030, 0.036, and 0.915, respectively, was superior to the BP, GA-BP, and RF-BP neural network models, and could better predict the moisture content of tea during the roasting process.

Predictive evaluation of dynamic responses and frequencies of bridge using optimized VMD and genetic algorithm-back propagation approach

Predictive evaluation of dynamic responses and frequencies of bridge using optimized VMD and genetic algorithm-back propagation approach

Predicting hospital admissions for upper respiratory tract complaints: An artificial neural network approach integrating air pollution and meteorological factors.

This study uses artificial neural networks (ANNs) to examine the intricate relationship between air pollutants, meteorological factors, and respiratory disorders. The study investigates the correlation between hospital admissions for respiratory diseases and the levels of PM10 and SO2 pollutants, as well as local meteorological conditions, using data from 2017 to 2019. The objective of this study is to clarify the impact of air pollution on the well-being of the general population, specifically focusing on respiratory ailments. An ANN called a multilayer perceptron (MLP) was used. The network was trained using the Levenberg-Marquardt (LM) backpropagation algorithm. The data revealed a substantial increase in hospital admissions for upper respiratory tract diseases, amounting to a total of 11,746 cases. There were clear seasonal fluctuations, with fall having the highest number of cases of bronchitis (N = 181), sinusitis (N = 83), and upper respiratory infections (N = 194). The study also found demographic differences, with females and people aged 18 to 65years having greater admission rates. The performance of the ANN model, measured using R2 values, demonstrated a high level of predictive accuracy. Specifically, the R2 value was 0.91675 during training, 0.99182 during testing, and 0.95287 for validating the prediction of asthma. The comparative analysis revealed that the ANN-MLP model provided the most optimal result. The results emphasize the effectiveness of ANNs in representing the complex relationships between air quality, climatic conditions, and respiratory health. The results offer crucial insights for formulating focused healthcare policies and treatments to alleviate the detrimental impact of air pollution and meteorological factors.

Prediction of Short-Term Winter Photovoltaic Power Generation Output of Henan Province Using Genetic Algorithm–Backpropagation Neural Network

Prediction of Short-Term Winter Photovoltaic Power Generation Output of Henan Province Using Genetic Algorithm–Backpropagation Neural Network

Study on the Application of Error Back-Propagation Algorithm Applied to the Student Status Management in Higher Education Institutions

The objective of this work is to predict the employment rate of students based on the information in the SSM (student status management) in colleges and universities. Firstly, the relevant content of SSM is introduced. Secondly, the BP (Back Propagation) neural network, the LM (Levenberg Marquardt) algorithm, and the BR (Bayesian Regularization) algorithm are introduced. In addition, the LM algorithm is combined with the BR algorithm to optimize the BP neural network, so as to establish a prediction model of the employment rate of college graduates based on the LM-BP neural network (the established prediction model). Finally, the established prediction model is verified after the historical data of college students in the SSM are managed and processed using the big data analysis technology. It suggests that the established prediction model shows higher prediction accuracy, more stable prediction performance, more ideal prediction effect, and higher practical application value.

Artificial intelligence and machine learning in optics: tutorial

Across the spectrum of scientific inquiry and practical applications, the emergence of artificial intelligence (AI) and machine learning (ML) has comprehensively revolutionized problem-solving methodologies. This tutorial explores key aspects of AI/ML and their remarkable role in augmenting the capabilities of optics and photonics technologies. Beginning with fundamental definitions and paradigms, the tutorial progresses to classical machine learning algorithms, with examples employing support vector machines and random forests. Extensive discussion of deep learning encompasses the backpropagation algorithm and artificial neural networks, with examples demonstrating the applications of dense and convolutional neural networks. Data augmentation and transfer learning are examined next as effective strategies for handling scenarios with limited datasets. Finally, the necessity of alleviating the burden of data collection and labeling is discussed, motivating the investigation of unsupervised and semi-supervised learning strategies as well as the utilization of reinforcement learning. By providing a structured exploration of AI/ML techniques, this tutorial equips researchers with the essential tools to begin leveraging AI’s transformative potential within the expansive realm of optics and photonics.

Data Mining Forecasting Methods for Predicting the Availability of Sustainable Rice Commodities in Bali Province

This research aims to prepare the capacity of local governments to provide sufficient strategic food production, especially rice, which is the commodity most consumed by the people of Gianyar Regency every year. This research is descriptive research with a quantitative approach. The data used in the analysis is secondary data in the form of rice production data (tons), rice production (tons), harvested area (ha), public consumption of rice commodities (kg) and population (people) from 2018 to 2022. obtained from the Central Statistics Agency (BPS) and the Bali Province Agriculture and Food Security Service. The results of this research show that the forecasting method using the Backpropagation algorithm can be used to provide an approach to forecasting rice harvest area, rice production, rice production, community consumption levels, and population in 2023 and 2024 in Gianyar Regency. The Backpropagation architectural design uses 12 input layers, 10 hidden layers, and 1 output with Mean Absolute Percentage Error (MAPE) results of 0.22 and Mean Absolute Deviation (MAD) of 339 with a very good forecasting category. The average ratio of normative per capita consumption to food availability, especially rice commodities in Gianyar Regency in 2023 will experience a high deficit condition with a Cnorm Ratio value (z) = 1.52 and in 2024 will experience a high deficit condition with a Cnorm Ratio value (z) = 1.57. The forecasting method using the Backpropagation Algorithm can still be developed by focusing on each district by calculating the overall food commodities in that district.

Hardware implementation of backpropagation using progressive gradient descent for in situ training of multilayer neural networks.

Neural network training can be slow and energy-expensive due to the frequent transfer of weight data between digital memory and processing units. Neuromorphic systems can accelerate neural networks by performing multiply-accumulate operations in parallel using nonvolatile analog memory. However, executing the widely used backpropagation training algorithm in multilayer neural networks requires information-and therefore storage-of the partial derivatives of the weight values preventing suitable and scalable implementation in hardware. Here, we propose a hardware implementation of the backpropagation algorithm that progressively updates each layer using in situ stochastic gradient descent, avoiding this storage requirement. We experimentally demonstrate the in situ error calculation and the proposed progressive backpropagation method in a multilayer hardware-implemented neural network. We confirm identical learning characteristics and classification performance compared to conventional backpropagation in software. We show that our approach can be scaled to large and deep neural networks, enabling highly efficient training of advanced artificial intelligence computing systems.

Gradient-free training of recurrent neural networks using random perturbations.

Recurrent neural networks (RNNs) hold immense potential for computations due to their Turing completeness and sequential processing capabilities, yet existing methods for their training encounter efficiency challenges. Backpropagation through time (BPTT), the prevailing method, extends the backpropagation (BP) algorithm by unrolling the RNN over time. However, this approach suffers from significant drawbacks, including the need to interleave forward and backward phases and store exact gradient information. Furthermore, BPTT has been shown to struggle to propagate gradient information for long sequences, leading to vanishing gradients. An alternative strategy to using gradient-based methods like BPTT involves stochastically approximating gradients through perturbation-based methods. This learning approach is exceptionally simple, necessitating only forward passes in the network and a global reinforcement signal as feedback. Despite its simplicity, the random nature of its updates typically leads to inefficient optimization, limiting its effectiveness in training neural networks. In this study, we present a new approach to perturbation-based learning in RNNs whose performance is competitive with BPTT, while maintaining the inherent advantages over gradient-based learning. To this end, we extend the recently introduced activity-based node perturbation (ANP) method to operate in the time domain, leading to more efficient learning and generalization. We subsequently conduct a range of experiments to validate our approach. Our results show similar performance, convergence time and scalability when compared to BPTT, strongly outperforming standard node perturbation and weight perturbation methods. These findings suggest that perturbation-based learning methods offer a versatile alternative to gradient-based methods for training RNNs which can be ideally suited for neuromorphic computing applications.

Maximum displacement prediction model for steel beams with hexagonal web openings under impact loading based on artificial neural networks

The estimation of the maximum displacement of steel beams with hexagonal web openings under impact loads is crucial for the anti-impact design of structures. The dynamic characteristics are complex, and the maximum displacement can be obtained experimentally, although the time cost, and expense of such experiments are high. In this context, a 6-5-1 artificial neural network model based on the Levenberg-Marquardt backpropagation algorithm was developed, incorporating 5-fold cross-validation techniques. Due to limited experimental data, 324 high-precision finite element models were created in bulk using Python-based ABAQUS software scripts to provide additional numerical data, and 26 specimens were designed for drop hammer tests to validate the finite element models. The artificial neural network model on the test data yielded a mean squared error of 0.0268, a mean absolute error of 0.0014, and a coefficient of determination value of 0.9806, with samples having an error of less than 10% comprising 77.16% of the total samples. Using Garson’s algorithm, the contribution of input features was estimated, and a full parameter analysis of the proposed model was conducted. The results indicate that the most significant factor is the impact mass, which accounts for 37.58%. When the impact mass increased from 530 kg to 630 kg, the maximum displacement of Q235 surged by 93.75%. The factor with the least impact is the opening spacing, accounting for 2%, with an average increase in maximum displacement of only 3.45%. Finally, ANN-based equations that can be used as design tools are established.

Classification of Malignancy of Lung Cancer Using Backpropagation Algorithm on CT-Scan Images

In this study, we investigate the classification of lung cancer CT scan images based on malignancy level using a backpropagation artificial neural network (ANN). Lung cancer is a deadly disease characterized by the growth of abnormal lung cells. The proposed method involves preprocessing to enhance image quality, followed by feature extraction using the Gray Level Co-occurrence Matrix (GLCM) method with angle variations of 0°, 45°, 90°, 135°, and d=1. The extracted features include energy, contrast, correlation, and homogeneity. The energy value range in malignant cancer is 0.27 to 0.81, while in benign cancer it is 0.26 to 0.73. The contrast in benign cancer ranges from 1.38 to 11.87, while in malignant cancer it is 1.47 to 13.67. The image correlation for malignant cancer is between 0.63 to 0.94, while for benign cancer it is 0.69 to 0.96. Homogeneity in malignant cancer has a value range between 0.67 to 0.91, while in benign cancer it ranges from 0.70 to 0.92. The classification of lung cancer malignancy is restricted to benign and malignant levels using a network architecture of [4 10 2], maximum iteration of 100000, and learning rate of 0.001. The accuracy of the testing data from the ANN is between 90% and 100%. These results demonstrate the effectiveness of the GLCM method and backpropagation algorithm in accurately classifying the malignancy level of lung cancer, which could aid in the early detection and treatment of the disease.

Novel online portfolio selection algorithm using deep sequence features and reversal information

Computational finance combines machine learning with financial needs to provide more efficient solutions for investment analysis and automated trading. In previous studies, traditional online portfolio selection (OLPS) algorithms were found to be overly reliant on artificially designed, subjective financial features. To address this issue, we propose a new predictive price tracking algorithm based on deep sequence features and reversal information (DSF-RI-PPT) for OLPS, extending a hybrid stock prediction algorithm to a multi-asset trading algorithm. We respectively employ the complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN), principal component analysis (PCA) algorithms and long short-term memory (LSTM) network to perform decomposition, feature extraction and prediction on financial data. Further, we supplement the reversal information by modifying the predicted prices with a reversal indicator-rate of change (ROC). Finally, we introduce a fast error back-propagation algorithm to integrate the predictive information into the investment ratio using gradient projection. Through empirical comparison and statistic analysis of the DSF-RI-PPT algorithm, price-tracking algorithms with similar prediction models, and nine classic OLPS algorithms in nine portfolio data sets under three financial indexes, it can be found that the DSF-RI-PPT algorithm is profitable and generalizable.

Calcium carbonate scale thickness prediction in annular three-phase flow using gamma-ray densitometry and artificial neural networks

This study presents a methodology based on gamma-ray densitometry for predicting calcium carbonate (CaCO3) scale thickness in pipelines of three-phase systems (oil, salty water, and gas) in the petroleum industry. Both concentric and eccentric scale thicknesses were calculated using analytical equations and artificial neural networks. The artificial neural networks were trained using a backpropagation algorithm with data acquired from the Monte Carlo N-Particle (MCNP6) computer code. The model utilized to achieve the supervised training of the network consisted of three 1¼ × ¾″ NaI(Tl) detectors positioned 120° apart around the pipe. Results demonstrate that the networks successfully predicted over 96 % of both concentric and eccentric cases of CaCO3 scale thickness with a relative error within 10%.