Error Backpropagation Method Research Articles

Large-Scale Computing Systems Workload Prediction Using Parallel Improved LSTM Neural Network

In recent years, large-scale computing systems have been widely used as an important part of the computing infrastructure. Resource management based on systems workload prediction is an effective way to improve application efficiency. However, accuracy and real-time functionalities are always the key challenges that perplex the systems workload prediction model. In this paper, we first investigate the dependence on historical workload in large-scale computing systems and build a day and time two-dimensional time-series workload model. We then design a two-dimensional long short-term memory (LSTM) neural network cell structure. Based on this, we propose an improved LSTM prediction model providing its mathematical description and an error back propagation method. Furthermore, to achieve systems resource management real-time requirement, we provide a parallel improved LSTM algorithm that uses a hidden layer week-based dependence and weights parallelization algorithm. The comparative studies, based on the actual workload of the Shanghai Supercomputer Center, demonstrate that our proposed improved LSTM neural network prediction model can achieve higher accuracy and real-time performance in large-scale computing systems.

3D Fourier Ptychographic Microscopy Based on the Beam Propagation Method and Time-Reversal Scheme

3D Fourier ptychographic microscopy can recover objects with high resolution across large volumes, which is very challenging in 3D imaging particularly because multiple scattering may occur within the sample. In this paper, we assume the thick sample is composed of a number of thin slices, and employ the beam propagation method (BPM) to model the propagation process of the light wave among successive slices capturing multiple scattering effects. 3D imaging is then accomplished by utilizing the gradient descent method to minimize the difference between the estimated intensity images by BPM and the captured measurements with angle-varied illuminations. We adopt a time-reversal scheme to obtain the gradient of the transmitted light intensity with respect to the complex refractive index of the volumetric sample and use the error backpropagation method to update the 3D sample iteratively. To further preserve the sharpness of the edges, we introduce a sparsity regularization term into the optimization process. Our method can achieve higher performance of 3D reconstruction compared to the original multi-slice approach, demonstrated in both simulation and experiment.

Prediction of Flow Accelerated Corrosion of NPP Pipeline Elements by Network Simulation Method

Based on a comprehensive approach that uses the computer simulation of the process of destroying structural materials and technology of self-learning neural networks, a methodology has been developed for predicting the rate of flow accelerated corrosion (FAC) of pipeline elements with a single-phase medium of the second circuit of nuclear power plants (NPPs). The neural network model has been implemented in the Delphi Integrated Development Environment. The neural network consists of an input layer containing seven elements and an output layer with two elements. As the input variables of the neural network, the parameters that have the greatest influence on FAC process are chosen. These are the medium temperature, the pipeline internal diameter, the oxygen content in the medium, the coolant flow velocity, the hydrogen index, the time of monitoring (or the start of operation), and the time for which the prediction is performed. For each of the network input parameters, intervals of possible values were chosen. At that, the factors that affect FAC rate, but not included in the feasible model (chromium, copper and molybdenum content in the pipeline material, amine type) are assumed to be permanent. As the output parameters of the neural network, FAC rate and the variation of the pipeline element wall thickness within the predicted time interval have been selected. As the activation function of the neural network the sigmoid function is used. As a method of training the neural network, the error back propagation method has been chosen, which assumes both a forward and reverse passage through the network layers. As the learning algorithm of the neural network, the one with a teacher has been chosen. As a test sample for the neural network, it is proposed, along with operational control data, to use the results of calculations based on a statistical model created in the framework of a special calculation-experimental method. The application of the developed methodology makes it possible to improve the prediction accuracy of FAC rate without determining all the dependencies between the many factors that influence FAC process. The low errors of the constructed models make it possible to use the results of calculations both to determine the resource characteristics of NPP pipelines and optimize operational control.

A review on neural networks with random weights

In big data fields, with increasing computing capability, artificial neural networks have shown great strength in solving data classification and regression problems. The traditional training of neural networks depends generally on the error back propagation method to iteratively tune all the parameters. When the number of hidden layers increases, this kind of training has many problems such as slow convergence, time consuming, and local minima. To avoid these problems, neural networks with random weights (NNRW) are proposed in which the weights between the hidden layer and input layer are randomly selected and the weights between the output layer and hidden layer are obtained analytically. Researchers have shown that NNRW has much lower training complexity in comparison with the traditional training of feed-forward neural networks. This paper objectively reviews the advantages and disadvantages of NNRW model, tries to reveal the essence of NNRW, gives our comments and remarks on NNRW, and provides some useful guidelines for users to choose a mechanism to train a feed-forward neural network.

Recognition of Actions of Medical Workers on the Basis of Accelerometer Readings with the Use of a Deep Belief Network

This paper analyzes a real set of large-volume medico-statistical data used for the recognition of actions of medical workers on the basis of accelerometer readings at definite moments of time. During the recognition process, a deep belief network was applied to unlabeled data, and then it was supervisedly learned using the error backpropagation method. The obtained results demonstrate higher recognition accuracy in comparison with basic methods. A considerable improvement in medical staff productivity is also achieved.

Neural network technique for identifying prognostic anomalies from low-frequency electromagnetic signals in the Kuril–Kamchatka region

In this paper, we suggest a technique for forecasting seismic events based on the very low and low frequency (VLF and LF) signals in the 10 to 50 Hz band using the neural network approach, specifically, the error back-propagation method (EBPM). In this method, the solution of the problem has two main stages: training and recognition (forecasting). The training set is constructed from the combined data, including the amplitudes and phases of the VLF/LF signals measured in the monitoring of the Kuril-Kamchatka region and the corresponding parameters of regional seismicity. Training the neural network establishes the internal relationship between the characteristic changes in the VLF/LF signals a few days before a seismic event and the corresponding level of seismicity. The trained neural network is then applied in a prognostic mode for automated detection of the anomalous changes in the signal which are associated with seismic activity exceeding the assumed threshold level. By the example of several time intervals in 2004, 2005, 2006, and 2007, we demonstrate the efficiency of the neural network approach in the short-term forecasting of earthquakes with magnitudes starting from M ≥ 5.5 from the nighttime variations in the amplitudes and phases of the LF signals on one radio path. We also discuss the results of the simultaneous analysis of the VLF/LF data measured on two partially overlapping paths aimed at revealing the correlations between the nighttime variations in the amplitude of the signal and seismic activity.

Improving artificial neural network model predictions of daily average PM10 concentrations by applying principle component analysis and implementing seasonal models

This study introduces a seasonal modeling approach in the prediction of daily average PM10 (particulate matter with an aerodynamic diameter <10 μm) levels 1 day ahead based on multilayer perceptron artificial neural network (MLP-ANN) forecasters. The data set covered all daily based meteorological parameters and PM10 concentrations in the period of 2007–2014. Seasonal ANN models for winter and summer periods were separately developed and trained by using a lagged time series data set. The most significant lagged terms of the variables within a 1-week period were determined by principal component analysis (PCA) and assigned as input vectors of ANN models. Cascading training with error back-propagation method was applied in model building. The use of seasonal ANN models with PCA-based inputs showed an increased prediction performance compared with nonseasonal models. For seasonal ANN models, the overall model agreement in training between modeled and observed values varied in the range of 0.78–0.83 and R2 values ranged in 0.681–0.727, which outperformed nonseasonal models. The best testing R2 values of seasonal models for winter and summer periods ranged in 0.709–0.727 with lower testing error, and the models did not show a tendency towards overpredicting or underpredicting the PM10 levels. The approach demonstrated in the study appeared to be promising for predicting short-term levels of pollutants through the data sets with high irregularities and could have significant applicability in the case of large number of considered inputs.Implications: This study provides an alternative approach to predict PM10 levels 1 day ahead by building seasonal ANN models. Applying PCA on a lagged data set resulted in selection of the most significant lags of variables reducing model complexity. Cascading training with error back-propagation method appropriately determined hidden layer neurons. Separately building ANN models for winter and summer periods over years, even though it required much more effort compared with building regular nonseasonal models, yielded better model agreements and smaller testing errors. This approach can be applied on the data sets with irregularities and a large number of considered inputs.

Decentralized PID neural network control for a quadrotor helicopter subjected to wind disturbance

A decentralized PID neural network (PIDNN) control scheme was proposed to a quadrotor helicopter subjected to wind disturbance. First, the dynamic model that considered the effect of wind disturbance was established via Newton-Euler formalism. For quadrotor helicopter flying at low altitude in actual situation, it was more susceptible to be influenced by the turbulent wind field. Therefore, the turbulent wind field was generated according to Dryden model and taken into consideration as the disturbance source of quadrotor helicopter. Then, a nested loop control approach was proposed for the stabilization and navigation problems of the quadrotor subjected to wind disturbance. A decentralized PIDNN controller was designed for the inner loop to stabilize the attitude angle. A conventional PID controller was used for the outer loop in order to generate the reference path to inner loop. Moreover, the connective weights of the PIDNN were trained on-line by error back-propagation method. Furthermore, the initial connective weights were identified according to the principle of PID control theory and the appropriate learning rate was selected by discrete Lyapunov theory in order to ensure the stability. Finally, the simulation results demonstrate that the controller can effectively resist external wind disturbances, and presents good stability, maneuverability and robustness.

Multilayer Neuro PID Controller based on Back Propagation Algorithm

Abstract The paper covers the basic idea, mathematical features of conventional Neuro PID controller which is a technique to make any linear or nonlinear system unaffected to unpredictability of system's parameters and disturbances such as noise. Here we suggest a technique to apply neural networks for the tuning of the PID (proportional, integral and derivative) controller's gains in a way human tune the gains depending on the environmental and systems requirements. Error back-propagation method is used as the tuning method for the controller which is also known as BP method and this method works on the local minima algorithm.

Pose-graph optimized displacement estimation for structural displacement monitoring

A visually servoed paired structured light system (ViSP) was recently proposed as a novel estimation method of the 6-DOF (Degree-Of-Freedom) relative displacement in civil structures. In order to apply the ViSP to massive structures, multiple ViSP modules should be installed in a cascaded manner. In this configuration, the estimation errors are propagated through the ViSP modules. In order to resolve this problem, a displacement estimation error back-propagation (DEEP) method was proposed. However, the DEEP method has some disadvantages: the displacement range of each ViSP module must be constrained and displacement errors are corrected sequentially, and thus the entire estimation errors are not considered concurrently. To address this problem, a pose-graph optimized displacement estimation (PODE) method is proposed in this paper. The PODE method is based on a graph-based optimization technique that considers entire errors at the same time. Moreover, this method does not require any constraints on the movement of the ViSP modules. Simulations and experiments are conducted to validate the performance of the proposed method. The results show that the PODE method reduces the propagation errors in comparison with a previous work.

Comparison of Artificial Neural Network Architecture in Solving Ordinary Differential Equations

This paper investigates the solution of Ordinary Differential Equations (ODEs) with initial conditions using Regression Based Algorithm (RBA) and compares the results with arbitrary- and regression-based initial weights for different numbers of nodes in hidden layer. Here, we have used feed forward neural network and error back propagation method for minimizing the error function and for the modification of the parameters (weights and biases). Initial weights are taken as combination of random as well as by the proposed regression based model. We present the method for solving a variety of problems and the results are compared. Here, the number of nodes in hidden layer has been fixed according to the degree of polynomial in the regression fitting. For this, the input and output data are fitted first with various degree polynomials using regression analysis and the coefficients involved are taken as initial weights to start with the neural training. Fixing of the hidden nodes depends upon the degree of the polynomial. For the example problems, the analytical results have been compared with neural results with arbitrary and regression based weights with four, five, and six nodes in hidden layer and are found to be in good agreement.

The displacement estimation error back-propagation (DEEP) method for a multiple structural displacement monitoring system

Visually servoed paired structured light system (ViSP) has been found to be useful in estimating 6-DOF relative displacement. The system is composed of two screens facing each other, each with one or two lasers, a 2-DOF manipulator and a camera. The displacement between two sides is estimated by observing positions of the projected laser beams and rotation angles of the manipulators. To apply the system to massive structures, the whole area should be partitioned and each ViSP module is placed in each partition in a cascaded manner. The estimated displacement between adjoining ViSPs is combined with the next partition so that the entire movement of the structure can be estimated. The multiple ViSPs, however, have a major problem that the error is propagated through the partitions. Therefore, a displacement estimation error back-propagation (DEEP) method which uses Newton–Raphson or gradient descent formulation inspired by the error back-propagation algorithm is proposed. In this method, the estimated displacement from the ViSP is updated using the error back-propagated from a fixed position. To validate the performance of the proposed method, various simulations and experiments have been performed. The results show that the proposed method significantly reduces the propagation error throughout the multiple modules.

Regression-based neural network training for the solution of ordinary differential equations

In this paper, we have introduced a method which is based on the use of unsupervised type of regression-based algorithm (RBA) for solving ordinary differential equations (ODEs) with initial or boundary conditions. Approximate solution of differential equation is differentiable and closed analytic. Here we have used error back propagation method for minimising the error function and modification of the parameters without direct use of other optimisation techniques. Initial weights are taken as combination of random as well as by proposed regression-based method. We present the method for solving a variety of problems and the results with arbitrary and regression-based initial weights are compared. Here the number of nodes in hidden layer has been fixed according to the degree of polynomial in the regression. The present model demonstrates to get also the approximate solutions for the differential equation inside and outside of the training domain.

A new method for position control of a 2-DOF robot arm using neuro–fuzzy controller

Robot manipulators have become increasingly important in the field of flexible automation. So modeling and control of robots in automation will be very important. But Robots, as complex systems, must detect and isolate faults with high probabilities while doing their tasks with humans or other robots with high precision and they should tolerate the fault with the controller.This paper introduces a Neuro-Fuzzy Controller (NFC) for position control of robot arm. A five layer neural network is used to adjust input and output parameters of membership function in a fuzzy logic controller. The hybrid learning algorithm is used for training this network. In this algorithm, the least square estimation method is applied for the tuning of linear output membership function parameters and the error backpropagation method is used to tune the nonlinear input membership function parameters. The simulation results show that NFC is better and more robust than the PID controller for robot trajectory control.

PV Maximum Power‐Point Tracking by Using Artificial Neural Network

In this paper, using artificial neural network (ANN) for tracking of maximum power point is discussed. Error back propagation method is used in order to train neural network. Neural network has advantages of fast and precisely tracking of maximum power point. In this method neural network is used to specify the reference voltage of maximum power point under different atmospheric conditions. By properly controling of dc‐dc boost converter, tracking of maximum power point is feasible. To verify theory analysis, simulation result is obtained by using MATLAB/SIMULINK.

An Automatic System Using Mobile-Agent Software to Model the Calculation Process of a Chemical Vapor Deposition Film Deposition Simulator

We have developed an automatic modeling system for calculation processes of the simulator to reproduce experimental results of chemical vapor deposition (CVD), in order to decrease the calculation cost of the simulator. Replacing the simulator by the mathematical models proposed by the system will contribute towards decreasing the calculation costs for predicting the experimental results. The system consists of a mobile agent and two software resources in computer networks, that is, generalized modeling software and a simulator reproducing cross-sections of the deposited films on the substrates with the micrometer- or nanometer-sized trenches. The mobile agent autonomously creates appropriate models by moving to and then operating the software resources. The models are calculated by partial least squares regression (PLS), quadratic PLS (QPLS) and error back propagation (BP) methods using artificial neural networks (ANN) and expresses by mathematical formulas to reproduce the calculated results of the simulator. The models show good reproducibility and predictability both for uniformity and filling properties of the films calculated by the simulator. The models using the BP method yield the best performance. The filling property data are more suitable to modeling than film uniformity.

Scaling-efficient in-situ training of CMOL CrossNet classifiers

CMOL CrossNets, hybrid CMOS/nanoelectronic neuromorphic circuits, may open up exciting opportunities to build artificial intelligence similar to the brain. However, limited functionality of nanodevices used in CMOL circuits causes significant challenges to train CrossNets with the usual algorithms. In order to overcome these challenges, we developed an in-situ variety of the error backpropagation method for supervised training of CrossNet-based pattern classifiers. Although this algorithm successfully trained CrossNets to perform simple benchmark classification tasks in Proben1, we found that it did not scale up to larger problems such as the MNIST dataset. Therefore, we propose an alternative in-situ method, combining training with the hidden layer build-up. Simulated results suggest that our new in-situ approach is appropriate to train CrossNets to perform classification on practical problems.

An Automatic Modeling System of the Calculation Process of a CVD Film Deposition Simulator

We have developed a modeling system for the calculation processes of a process simulator for chemical vapor deposition (CVD). The system consists of a software agent, generalized modeling software and a simulator reproducing cross-sections of the deposited films. The agent autonomously creates appropriate models by operating the simulator and the modeling software. The models are calculated by partial least squares regression (PLS), quadratic PLS (QPLS) and error back propagation (BP) methods using artificial neural networks (ANN) and expresses by mathematical formulas to reproduce the calculated results of the simulator. In addition, we have investigated the performance of the models. The models show good reproducibility and predictability both for uniformity and filling properties of the films calculated by the simulator. The models using the BP method yield the best performance. The filling property data are more suitable to modeling than film uniformity. Since the simulator can be replaced by the models, the models will contribute towards decreasing the calculation costs for predicting the experimental results. This study thus presents an example of creating a robust and reliable predictor as a virtual reactor yielding experimental results that is expected to be free of the difficulties of extrapolation, using ANN and other mathematical models.

Electronic Nose for a Fire Detection System by Neural Networks

Conventional fire detectors use the smoke density or the high air temperature to trigger the fire alarm. These devices lack of ability to detect the source of fire in the early stage and they always create false alarms. In this paper, a reliable electronic nose system designed from the combination of various metal oxide gas sensors is applied to detect the early stage of fire from various sources. The time series signals of the same source of fire in every repetition data are highly correlated and each source of fire has a unique pattern of time series data. Therefore, the error back propagation method can classify the tested smell with 99.6% of correct classification by using only a single training data from each source of fire. The use of k -means algorithms results in 98.3% of correct classification which also shows the high ability of the electronic nose to detect the early stage of fire from various sources accurately.

Efficient text independent speaker recognition with wavelet feature selection based multilayered neural network using supervised learning algorithm

A wavelet packet feature selection derived by using multilayered neural network for speaker identification is described. The concept of a multilayered neural network is without using a gradient method. First, the outputs of each hidden unit are algebraically determined by an error backpropagation method. Then, the weight parameters are determined by using an exponentially weighted least squares method. Our results have shown that this feature selection introduced better performance than the other methods with respect to the percentages of recognition.