Classical Backpropagation Research Articles

Two-stage fusion framework driven by domain knowledge for penetration prediction of laser welding

To solve the problem of industrial application difficulties caused by insufficient depth exploration and poor interpretability of the pure data drive neural network. A two-stage fusion framework driven by domain knowledge was proposed and focused on industrial applications: laser welding penetration monitoring. First, a multi-sensor acquisition system based on plasma and molten pool signals is developed, extracting multiple features from plasma and molten pool images. Two Back Propagation (BP) neural network information fusion models are established in the first fusion stage, leveraging the characteristics of the plasma plume and molten pool front. In the second fusion stage, the Dempster-Shafer (DS) evidence theory is employed to fuse the information from the two BP neural network models. The proposed method, achieving a high accuracy of 98.7 %, is validated by comparing its average accuracy and anti-interference capability with classical Convolutional Neural Network (CNN) and BP neural network models. The physical relationship between input and output is elucidated through an established physical model, enhancing the interpretability and anti-interference capability of the model, and thereby improving the stability and accuracy of the system.

A Study on Speech Recognition by a Neural Network Based on English Speech Feature Parameters

In this study, from the perspective of English speech feature parameters, two feature parameters, the mel-frequency cepstral coefficient (MFCC) and filter bank (Fbank), were selected to identify English speech. The algorithms used for recognition employed the classical back-propagation neural network (BPNN), recurrent neural network (RNN), and long short-term memory (LSTM) that were obtained by improving RNN. The three recognition algorithms were compared in the experiments, and the effects of the two feature parameters on the performance of the recognition algorithms were also compared. The LSTM model had the best identification performance among the three neural networks under different experimental environments; the neural network model using the MFCC feature parameter outperformed the neural network using the Fbank feature parameter; the LSTM model had the highest correct rate and the highest speed, while the RNN model ranked second, and the BPNN model ranked worst. The results confirm that the application of the LSTM model in combination with MFCC feature parameter extraction to English speech recognition can achieve higher speech recognition accuracy compared to other neural networks.

Spike-based computation using classical recurrent neural networks

Spiking neural networks (SNNs) are a type of artificial neural networks in which communication between neurons is only made of events, also called spikes. This property allows neural networks to make asynchronous and sparse computations and therefore drastically decrease energy consumption when run on specialized hardware. However, training such networks is known to be difficult, mainly due to the non-differentiability of the spike activation, which prevents the use of classical backpropagation. This is because state-of-the-art SNNs are usually derived from biologically-inspired neuron models, to which are applied machine learning methods for training. Nowadays, research about SNNs focuses on the design of training algorithms whose goal is to obtain networks that compete with their non-spiking version on specific tasks. In this paper, we attempt the symmetrical approach: we modify the dynamics of a well-known, easily trainable type of recurrent neural network (RNN) to make it event-based. This new RNN cell, called the spiking recurrent cell, therefore communicates using events, i.e. spikes, while being completely differen-tiable. Vanilla backpropagation can thus be used to train any network made of such RNN cell. We show that this new network can achieve performance comparable to other types of spiking networks in the MNIST benchmark and its variants, the Fashion-MNIST and the Neuromorphic-MNIST. Moreover, we show that this new cell makes the training of deep spiking networks achievable.

Artificial Neural Network based prediction of Specific Fuel Consumption in various Compression Ratio CI Engine Fueled with Various Plastic Composition Pyrolysis oil and Diesel

This study looks at the usage of Artificial Neural Network Modeling for predicting Specific Fuel Consumption for compression ignition engines running on Diesel, Low-Density Polyethylene Pyrolysis Oil (LDPE PO), High-Density Polyethylene Pyrolysis Oil (HDPE PO), and Polypropylene Pyrolysis Oil (PP PO). Using preliminary practical findings, The Model of ANN was built to estimate SFC by adjusting the parameters of the engine. SFC is anticipated by adjusting the parameters and utilizing the orthogonal array. The experiment is then carried out using an orthogonal array. The outcomes of experimental work are used to create an ANN model. In the case of a non-linear mapping between input & output, the classic back-propagation method and a multi-layer perception network are utilized. The values of Mean Square Error (MSE), Root Mean Square Error (RMSE), and Regression Coefficient (R2) for LM10TP architecture are 1.5143×10–06, 0.0012 & 1 in training & the validation values 1.2185×10–06, 0.0011 and 0.9999 respectively. It was discovered that neural networks are useful tools for prediction since they performed well in specific fuel consumption prediction in both training and validation.

Adaptive Grey Wolf Optimization Technique for Stock Index Price Prediction on Recurring Neural Network Variants

In this paper, we propose a Long short-term memory (LSTM) and Adaptive Grey Wolf Optimization (GWO)--based hybrid model for predicting the stock prices of the Major Indian stock indices, i.e., Sensex. The LSTM is an advanced neural network that handles uncertain, nonlinear, and sequential data. The challenges are its weight and bias optimization. The classical backpropagation has issues of dangling on local minima or overfitting the dataset. Thus, we propose a GWO-based hybrid approach to evolve the weights and biases of the LSTM and the dense layers. We have made the GWO more robust by introducing an approach to improve the best possible solution by using the optimal ranking of the wolves. The proposed model combines the GWO with Adam Optimizer to train the LSTM. Apart from the LSTM, we have also implemented the Adaptive GWO on other variants of Recurring Neural Networks (RNN) like LSTM, Bi-Directional LSTM, Gated Recurrent Units (GRU), and Bi-Directional GRU and computed the corresponding results. The Adaptive GWO here evolves the initial weights and biases of the above-discussed neural networks. In this research, we have also compared the forecasting efficiency of our proposed work with a particle-warm optimization (PSO) based hybrid LSTM model, simple Grey-wolf Optimization (GWO), and Adaptive PSO. According to the experimental findings, the suggested model has effectively used the best initial weights, and its results are the best overall.

Detection of arc grounding fault based on the features of fault voltage

Arc grounding faults commonly occur in power grids, causing extensive power outages and significant economic losses. In this paper, we propose a novel method for detecting arc grounding faults that utilizes a time-frequency-phase mixed feature extraction approach. The mathematical expression for the arc grounding fault voltage is detailed in this paper, and from this, time-domain features that are unaffected by fault transition resistance are extracted. Furthermore, features with clear physical interpretation are extracted from the frequency and phase domains of the fault signal. Our method is able to avoid the subjective feature selection problem that exists in current methods. We conduct fault detection experiments using a topology designed for the purpose and a 10 kV setup to obtain voltage data of arc grounding faults. We then train a fault recognition model using a classic backpropagation neural network. Experimental results demonstrate that the proposed method achieves an accuracy of 98.22% in identifying arc grounding faults. Additionally, we apply this method to identify corona discharge and surface discharge, achieving accuracies of 90.68% for corona discharge and 92.9% for surface discharge.

Modified Backpropagation Algorithm with Multiplicative Calculus in Neural Networks

Backpropagation is one of the most widely used algorithms for training feedforward deep neural networks. The algorithm requires a differentiable activation function and it performs computations of the gradient proceeding backwards through the feedforward deep neural network from the last layer through to the first layer. In order to calculate the gradient at a specific layer, the gradients of all layers are combined via the chain rule of calculus. One of the biggest disadvantages of the backpropagation is that it requires a large amount of training time. To overcome this issue, this paper proposes a modified backpropagation algorithm with multiplicative calculus. Multiplicative calculus provides an alternative to the classical calculus and it defines new kinds of derivative and integral forms in multiplicative form rather than addition and subtraction forms. The performance analyzes are discussed in various case studies and the results are given comparatively with classical backpropagation algorithm. It is found that the proposed modified backpropagation algorithm converges in less time to the solution and thus provides fast training in the given case studies. It is also shown that the proposed algorithm avoids the local minima problem.

Graphical multispectral radiation temperature inversion algorithm based on deep learning.

Neural networks are the most promising tool to solve the problem that an assumed emissivity model is needed in the field of multispectral radiometric temperature measurement. Existing neural network multispectral radiometric temperature measurement algorithms have been investigating the problems of network selection, network porting, and parameter optimization. The inversion accuracy and adaptability of the algorithms have been unsatisfactory. In view of the great success of deep learning in the field of image processing, this Letter proposes the idea of converting one-dimensional multispectral radiometric temperature data into two-dimensional image data for data processing to improve the accuracy and adaptability of multispectral radiometric temperature measurement by deep learning algorithms. Simulation and experimental validation are carried out. In the simulation, the error is less than 0.71% without noise and 1.80% with 5% random noise, which improves the accuracy by more than 1.55% and 2.66% compared with the classical BP (backpropagation) algorithm, and 0.94% and 0.96% compared with the GIM-LSTM (generalized inverse matrix-long short-term memory) algorithm. In the experiment, the error is less than 0.83%. This indicates that the method has high research value and is expected to lead multispectral radiometric temperature measurement technology to a new level.

A Dendritic Neuron Model Optimized by Meta-Heuristics with a Power-Law-Distributed Population Interaction Network for Financial Time-Series Forecasting

The famous McCulloch–Pitts neuron model has been criticized for being overly simplistic in the long term. At the same time, the dendritic neuron model (DNM) has been shown to be effective in prediction problems, and it accounts for the nonlinear information-processing capacity of synapses and dendrites. Furthermore, since the classical error back-propagation (BP) algorithm typically experiences problems caused by the overabundance of saddle points and local minima traps, an efficient learning approach for DNMs remains desirable but difficult to implement. In addition to BP, the mainstream DNM-optimization methods include meta-heuristic algorithms (MHAs). However, over the decades, MHAs have developed a large number of different algorithms. How to screen suitable MHAs for optimizing DNMs has become a hot and challenging area of research. In this study, we classify MHAs into different clusters with different population interaction networks (PINs). The performance of DNMs optimized by different clusters of MHAs is tested in the financial time-series-forecasting task. According to the experimental results, the DNM optimized by MHAs with power-law-distributed PINs outperforms the DNM trained based on the BP algorithm.

ResNet and PolyNet Based Identification and (MPC) Control of Dynamical Systems: A Promising Way

This paper deals with model predictive control synthesis which take benefits from artificial neural networks to model (non-linear) dynamical system. More precisely, thanks to a systematic and rigorous methodology, it is shown that residual networks (ResNet) and PolyInception networks (PolyNet) neural network architectures, developed initially for image recognition, are very good candidate for i) identification of dynamical systems, ii) being used as embedded model in a model predictive control laws. Concretely, the widely used non-linear dynamical system quadruple tank process is used as a benchmark. The neural network architectures studied are i) feedforward networks as a reference point, and the two other linked to Euler integration method ii) residual networks and iii) PolyInception networks. Networks training is performed by mixing classical back-propagation algorithm and hyperparameters optimisation through heuristics. The identification results provided show that neural networks of types ii) and iii) perform better than the classical one i), with a better generalisation capability. Finally, model predictive controllers are synthesized based on the various networks trained. The simulation results obtained for controlling water levels of a 4 tanks system benchmark give interesting insights. They show that residual networks based model predictive control is better suited than feedforward networks and PolyInception networks based ones, both taking into account computation time and set point errors.

An Aeromagnetic Compensation Algorithm Based on Radial Basis Function Artificial Neural Network

Aeromagnetic exploration is a magnetic exploration method that detects changes of the earth’s magnetic field by loading a magnetometer on an aircraft. With the miniaturization of magnetometers and the development of unmanned aerial vehicles (UAV) technology, UAV aeromagnetic surveying plays an increasingly important role in mineral exploration and other fields due to its advantages of low cost and safety. However, in the process of aeromagnetic measurement data, due to the ferromagnetic material of the aircraft itself and the change of flight direction and attitude, magnetic field interference will occur and affect the measurement of the geomagnetic field by the magnetometer. The work of aeromagnetic compensation is to compensate for this part of the magnetic interference and improve the magnetic measurement accuracy of the magnetometer. This paper focused on the problems of UAV aeromagnetic survey data processing and improved the accuracy of UAV based aeromagnetic data measurement. Based on the Tolles–Lawson model, a numerical simulation experiment of magnetic interference of UAV-based aeromagnetic data was carried out, and a radial basis function (RBF) artificial neural network (ANN) algorithm was proposed for the first time to compensate the aeromagnetic data. Compared with classical backpropagation (BP) ANN, the test results of the synthetic data and real measured magnetic data showed that the RBF-ANN has higher compensation accuracy and stronger generalization ability.

Optimized prediction models for faulting failure of Jointed Plain concrete pavement using the metaheuristic optimization algorithms

This study aims to predict faulting failure of jointed plain concrete pavement (JPCP) using different variables. For this purpose, four feature selection methods were developed by combining the artificial neural networks (ANN) and four multi-objective metaheuristic optimization algorithms, namely, the Pareto envelope-based selection algorithm II (PESA-2), the strength Pareto evolutionary algorithm 2 (SPEA-2), multi-objective particle swarm optimization (MPSO), and multi-objective evolutionary algorithm based on decomposition (MOEA/D). The MOEA/D showed better performance compared to the other models, which identified 17 input variables affecting faulting failure. In the next step, the classic back-propagation (BP), Biogeography-based optimization (BBO), invasive weed optimization (IWO), and simulated annealing algorithm (SAA) were combined with the ANN to develop three prediction models for faulting failure. Modeling with metaheuristic optimization algorithms showed better performance than the ordinary ANN. The pavement age, cumulative average precipitation, and elasticity modulus of the concrete slab have the most significant impact on the formation and increase of faulting.

Rotating Single-Antenna Spoofing Signal Detection Method Based on IPNN.

The traditional carrier-phase differential detection technology mainly relies on the spatial processing method, which uses antenna arrays or moving antennas to detect spoofing signals, but it cannot be applied to static single-antenna receivers. Aiming at this problem, this paper proposes a rotating single-antenna spoofing signal detection method based on the improved probabilistic neural network (IPNN). When the receiver antenna rotates at a constant speed, the carrier-phase double difference of the real signal will change with the incident angle of the satellite. According to this feature, the classification and detection of spoofing signals can be realized. Firstly, the rotating single-antenna receiver collects carrier-phase values and performs double-difference processing. Then, we construct an IPNN model, whose smoothing factor can be adaptively adjusted according to the type of failure mode. Finally, we use the IPNN model to realize the classification and processing of the carrier-phase double-difference observations and obtain the deception detection results. In addition, in order to reflect that the method has a certain practical value, we simulate the spoofing scenario of satellite signals and effectively identify abnormal satellite signals according to the detection results of the inter-satellite differential combination. Actual experiments indicate that the detection accuracy of the proposed method for spoofing signals reaches 98.84%, which is significantly better than the classical probabilistic neural network (PNN) and back-propagation neural network (BPNN), and the method can be implemented in fixed base station receivers for the real-time detection of forwarding spoofing.

Fault Diagnosis to Nuclear Power Plant System Based on Time-Series Convolution Neural Network

Nuclear power plant (NPP) is a highly complex engineering system which has typical internal feedback and strong component coupling. With these features, most NPP systems have high risk of radioactive release, which makes it essential to perform fault detection (FD) to the NPP systems. To address this challenge, this paper proposes a FD mechanism named characteristic time-series convolutional neural network (CT-CNN) based on principal component analysis (PCA), time-series analysis, and convolutional neural network (CNN) mechanisms. First, the models of NPP FD system are formulated. Then, the PCA mechanism is applied to extract the features of the NPP system. Next, the time-series analysis and CNN approaches are applied to realize FD to the NPP system. With the above mechanisms, the proposed approach has not only shown strong stability and become adaptive to different data set, but also preserves both time and state characteristics of the NPP system. In experiment, it shows the proposed approach can achieve better performance in both detection accuracy and variance than the classic back propagation, LSTM method, and standard CNN algorithms. More significantly, its optimal accuracy can be as high as 99.8%.

Particle Swarm-Assisted Artificial Neural Networks for Making Liquefied Natural Gas Processes Feasible Under Varying Feed Conditions

Natural gas (NG) has been widely recognized as a cleaner fuel compared to other fossil fuels. Reserves of NG are typically located in remote areas, and their conditions and compositions vary geographically. The NG from such areas is transported in the form of liquefied natural gas (LNG). Liquefying NG is highly complex. Generally, the process is designed to be carried out under fixed natural gas (NG) conditions; hence, it may not perform well under variable NG conditions. Considering this issue, the use of an artificial intelligence approach, rather than the conventional optimization one, was investigated to make the LNG process feasible under variable NG conditions. This study is the first in this research area to train an artificial neural network (ANN) using the particle swarm optimization (PSO) algorithm as a learning method. The developed PSO-ANN model was used to predict the decision variables of a single mixed refrigerant (SMR) LNG process for its feasible design under varying NG conditions. The correctness of the predicted set of decision variables (NG conditions) was verified by inputting them into Aspen HYSYS. The output of the SMR-LNG process was the overall power at a constrained minimum internal temperature approach (MITA) value, i.e., 1.0 ≤ MITA ≤3.0. The prediction results of the PSO-ANN model were compared with those of the classical ANN backpropagation learning method. The success rate of the proposed PSO-ANN model was 80%. Furthermore, the proposed model can make the LNG process feasible for a diverse range of temperature and pressure values. A feasible process with a better MITA value can also be achieved by tuning the model parameters.

Proportional-integral-derivative controller optimization by particle swarm optimization and back propagation neural network for a parallel stabilized platform in marine operations

The parallel stabilized platform is an important equipment to ensure the stability of marine operations, which can effectively mitigate collisions of lifted goods. The 3UPU_UP parallel stabilized platform system is proposed, and the kinematics and dynamics of the 3UPU_UP parallel stabilized platform are researched, where U denotes universal joint; P denotes prismatic joint. The platform motion can be realized by controlling the actuators to realize compensation ship motion. While conventional control algorithms are difficult to cope with complex nonlinear systems, the optimization control method has been proposed. Based on the back propagation (BP) neural network proportional-integral-derivative (PID) controller, the particle swarm optimization (PSO) is combined with the BP neural network, and the combined PSO-BP PID controller optimization algorithm is introduced into the servo control system. Compared with classic PID and BP PID, the overshoot of PSO-BP PID to the response of step signal is reduced by 10.2% and 11.48%, respectively, and 0.36s and 0.17s reduce the time required to reach stable states. By comparing the reduction of the error under various sea conditions, the upper platform tracking error curve is flatter under a PID controller based on PSO-BP, the tracking motion is closer to the theoretical value and the control accuracy is improved. The maximum tracking error of the upper platform is smaller, the overall motion vibration of the platform is reduced, and the anti-interference ability is enhanced. The PSO-BP PID controller for wave compensated platform servo system control is more reasonable, more adaptable to the complex and changing environment.

Intelligent Monitoring System Based on Noise-Assisted Multivariate Empirical Mode Decomposition Feature Extraction and Neural Networks.

Because of the nonlinearity and nonstationarity in the vibration signals of some rotating machinery, the analysis of these signals using conventional time- or frequency-domain methods has some drawbacks, and the results can be misleading. In this paper, a couple of features derived from multivariate empirical mode decomposition (MEMD) are introduced, which overcomes the shortcomings of the traditional features. A wind turbine gearbox and its bearings are investigated as rotating machinery. In this method, two types of feature structures are extracted from the decomposed signals resulting from the MEMD algorithm, called intrinsic mode function (IMF). The first type of feature vector element is the energy moment of effective IMFs. The other type of vector elements is amplitudes of a signal spectrum at the characteristic frequencies. A correlation factor is used to detect effective IMFs and eliminate the redundant IMFs. Since the basic MEMD algorithm is sensitive to noise, a noise-assisted extension of MEMD, NA-MEMD, is exploited to reduce the effect of noise on the output results. The capability of the proposed feature vector in health condition monitoring of the system is evaluated and compared with traditional features by using a discrimination factor. The proposed feature vector is utilized in the input layer of the classical three-layer backpropagation neural network. The results confirm that these features are appropriate for intelligent fault detection of complex rotating machinery and can diagnose the occurrence of early faults.

A Channeled Multilayer Perceptron as Multi-Modal Approach for Two Time-Frames Algo-Trading Strategy

FOREX (Foreign Exchanges) is a 24H open market with an enormous daily volume. Most of the used Trading strategies, used individually, are not providing accurate signals. In this paper, we are proposing an automated trading strategy that fits random market behaviors. It is based on neural networks applying triple exponential weighted moving average (EMA) as a trend indicator, Bollinger bands as a volatility indicator, and stochastic RSI as a momentum reversal indicator to prevent false indications in a short time frame. This approach is based on trend, volatility, and momentum reversal patterns combined with a market adaptive and a distributed multi-layer perceptron (MLP). It is called channeled multi-layer perceptron (CMLP) that is a neural network using channels and routines trained by previous profit/loss earned by triple EMA crossover, Bollinger Bands, and Stochastic RSI signals. Instead of using classic computations and Back-propagation for adjusting MLP parameters, we established a channeled multi-layer perceptron inspired by a multi-modal learning approach where each group of modalities (Channel) has its K_c That stands for a dynamic channel coefficient to produce a multi-processed feed-forward neural network that prevents uncertain trading signals depending on trend-volatility-momentum random patterns. CMLP has been compared to Multi-Modal GARCH-ARIMA and has proven its efficiency in unstable markets.

Application of Big Data Unbalanced Classification Algorithm in Credit Risk Analysis of Insurance Companies

The 2008 global financial crisis triggered by subprime mortgage crisis in the United States and the ongoing European debt crisis have urged governments and academics to pay high attention to financial industry risk supervision. The financial industry has actively implemented comprehensive risk management. As an important component of the financial industry, the insurance industry implements comprehensive risk management to control the risks of insurance companies. Propose an integrated learning model based on imbalanced dataset resampling and apply it to UCI dataset (University of California Irvine). First, resampling technology is used to preprocess the unbalanced dataset to obtain a relatively balanced training set. Then, use the classic backpropagation neural network, classic k-nearest neighbor, and classic Naive Bayes three algorithms as the base classifier and use the Bagging strategy to get the ensemble learning model. In order to verify its effectiveness, F‐measure and G‐mean methods are used to measure the performance of the classifier. The subject mainly focuses on the classification of relevance vector machine (RVM) in two types of large‐scale datasets, imbalanced and balanced, and proposes solutions for these two types of data. This explains the effectiveness of the disequilibrium classification algorithm used in the risk analysis of insurance companies.

State Diagnosis of Elevator Control Transformer over Vibration Signal Based on MEA-BP Neural Network

Vibration analysis is considered as an effective and reliable nondestructive technique for monitoring the operation conditions of elevator control transformer. In the paper, a novel model using the Empirical Mode Decomposition (EMD), the empirical wavelet packet transform, the mind evolutionary algorithm (MEA), and the backpropagation (BP) neural network is proposed for elevator control transformer fault diagnosis. Firstly, the collected signal is smoothed by EMD, the intrinsic mode function (IMF) components with large noise are determined according to the correlation coefficient, the wavelet adaptive threshold denoising algorithm is used to process the noisy IMF components, and the IMF components before and after processing and its residual component are reconstructed to obtain the denoised signal. Then, the denoised signal is transformed by empirical wavelet packet transform to extract the energy ratio and energy entropy features in the wavelet packet coefficients. Finally, a fault diagnosis model composed of MEA and BP neural network is developed, which avoids the problems of premature convergence and poor diagnosis effect. The experimental results show that the proposed model has a remarkable performance with an average root mean square error of 0.00672 and the average diagnosis accuracy of 90.8%, which is better than classic BP neural network.