Backpropagation Learning Rule Research Articles

Hybrid approach for gas viscosity in Yemeni oil fields

The estimation of a gas viscosity experimentally is often difficult. So, accurate determination of gas viscosity has been the main challenge in a gas reservoir development. There are many correlations to estimate this property. Often time, the results of these correlations do not agree with experimental data, thereby causing a considerable amount of error. The difficulty of these correlations can be propagated simply by tuning against some experimental data using artificial intelligent model. Currently, the achievements of artificial neural networks (ANN) techniques alone to predict gas viscosity open the door to use the hybrid system. In this model, the Particle Swarm Optimization (PSO) algorithm is employed to search for optimal connection weighs and thresholds for the neural networks (NN), then the back-propagation learning rule and training algorithm is used to adjust the final weights. A total of about 868 data points obtained from the laboratory measurements of gas viscosity were used. The data include measured gas viscosity, specific gas gravity, temperature, pressure, molecular weight, pseudo-critical temperature and pressure and non-hydrocarbon components (H2S, CO2, and N2). The performance of the PSONN model is compared with performance of ANN and other empirical model to show the most general and accurate model for predicting gas viscosity. From the results of this study, we found that the PSONN model is more reliable and accurate with the absolute present relative (APRE) error and mean square error (RMS) of 2.76 and 5.49 respectively.

Biologically plausible local synaptic learning rules robustly implement deep supervised learning.

In deep neural networks, representational learning in the middle layer is essential for achieving efficient learning. However, the currently prevailing backpropagation learning rules (BP) are not necessarily biologically plausible and cannot be implemented in the brain in their current form. Therefore, to elucidate the learning rules used by the brain, it is critical to establish biologically plausible learning rules for practical memory tasks. For example, learning rules that result in a learning performance worse than that of animals observed in experimental studies may not be computations used in real brains and should be ruled out. Using numerical simulations, we developed biologically plausible learning rules to solve a task that replicates a laboratory experiment where mice learned to predict the correct reward amount. Although the extreme learning machine (ELM) and weight perturbation (WP) learning rules performed worse than the mice, the feedback alignment (FA) rule achieved a performance equal to that of BP. To obtain a more biologically plausible model, we developed a variant of FA, FA_Ex-100%, which implements direct dopamine inputs that provide error signals locally in the layer of focus, as found in the mouse entorhinal cortex. The performance of FA_Ex-100% was comparable to that of conventional BP. Finally, we tested whether FA_Ex-100% was robust against rule perturbations and biologically inevitable noise. FA_Ex-100% worked even when subjected to perturbations, presumably because it could calibrate the correct prediction error (e.g., dopaminergic signals) in the next step as a teaching signal if the perturbation created a deviation. These results suggest that simplified and biologically plausible learning rules, such as FA_Ex-100%, can robustly facilitate deep supervised learning when the error signal, possibly conveyed by dopaminergic neurons, is accurate.

Prediction of solid circulation rate in an internal circulating fluidized bed: An empirical and ANN approach

Internal Circulating Fluidized beds (ICFBs) are interested in various industries because of their higher thermal efficiency and high reaction rates. However, understanding the complex flow hydrodynamics inside an ICFB is challenging. Also, the experiments needed are expensive and time-consuming. Therefore this study aims to predict the solids circulation rate in an in-house ICFB (0.3 m internal diameter x 3.0 m height) using an empirical model and an Artificial Neural Network (ANN) technique. The solid circulation rates measured at different operating and design conditions using a high-speed video camera are utilized to develop the models mentioned earlier. A dimensionless approach and nonlinear regression models are adopted to derive the empirical model. The Analysis of Variance (ANOVA) technique calculates F-number and their corresponding probabilities (P-values). The ANN model is developed with four input variables: particle size, static bed height, gap height, and gas superficial velocity. Multi-layer Perception model (MLP) with the Feedforward Back Propagation learning rule is employed to build the ANN model. A single hidden layer with nine neurons predicted a close solid circulation rate with minimal error over the empirical model compared to experimental data. Further, the empirical and ANN model's predicting capability is tested against literature data.

Double Moving Average Control Chart for Autocorrelated Data

The assumption of normality and independence is necessary for statistical inference of control charts. Misleading results are obtained if the traditional control chart technique is applied on the auto-correlated data. When data is correlated, a time series model is employed to produce an optimum output. The objective is to create a new control chart methodology that takes the autocorrelation of observations into account. Charts of Moving Average, Exponentially Weighted and Cumulative Sum better perform in the existence of autocorrelation data for small and moderate changes. The proposed methodology is highly skilled and receptive to minor, moderate and major changes in the process. Propsed DMA chart increases efficiency of average run length (ARL) chart for moving average (MA) to detect the small to medium magnitude shifts in the mean. The simulation also demonstrates that the DMA chart with spans of w=10 and 15 generally performs well in terms of average run length (ARL) as compared to clasical MA. This research may be extended to a multivariate autocorrelated statistical process control, but it can also be used to recognise and categorise seven categories of traditional control chart patterns, such as Downward, Upward Shift, Normal Trend, Cyclic, Systematic patterns, Increasing and Decresing Trend. In order to identify and categorize a set of subclasses of abnormal patterns, this model (multivariate autocorrelated statistical process control chart) should employ a multilayer feed forward Artificial Neural Network (ANN) architecture controlled by a back-propagation learning rule.

Prediction of stress responses in goats: comparison of artificial neural network and multiple regression models

This study was conducted to determine if artificial neural networks (ANN) can be used to more accurately predict physiological stress responses in goats compared with statistical regression. Prediction models were developed for plasma cortisol and glucose concentrations, creatine kinase (CK) activity, neutrophil (N) and lymphocyte (L) counts, and N:L ratio as a function of time (0, 1, 2, 3, and 4 h; n = 16 goats per time) after a 2.5 h transportation (input 1) and stocking density (25 vs. 50 goats; input 2). However, input 2 was not included in the final models because density did not have a significant effect. The NeuralWorks Predict® software and SAS were used to develop ANN and regression models, respectively. Backpropagation (BP) and Kalman filter (KF) learning rules were used to develop nonparametric models. Correlations between predicted and observed values were better with ANN-BP (R values = 0.87, 0.67, 0.56, 0.27, 0.42, and 0.53) and ANN-KF (R values = 0.84, 0.67, 0.58, 0.27, 0.42, and 0.50) models for cortisol, glucose, CK, N, L, and N:L ratio, respectively, than with regression models (R values =0.85, 0.52, 0.27, 0.13, 0.31, and 0.12). The results showed that the ANN models can predict responses more robustly compared with statistical regression.

Neural Network Programming in Python

In this paper a basic introduction to neural networks is made. An emphasis is given on a two layer perceptron used extensively for function approximation. The backpropagation learning rule is than briefly introduced. A short introduction into Python programming language is made and a program for the perceptron design is written and discussed in some detail. The “neurolab” library is used for this purpose.

Quantitative structure-property relationship modeling of polar analytes lacking UV chromophores to charged aerosol detector response.

In this study, a quantitative structure-property relationship model was built in order to link molecular descriptors and chromatographic parameters as inputs towards CAD responsiveness. Aminoglycoside antibiotics, sugars, and acetylated amino sugars, which all lack a UV/vis chromophore, were selected as model substances due to their polar nature that represents a challenge in generating a CAD response. Acetone, PFPA, flow rate, data rate, filter constant, SM5_B(s), ATS7s, SpMin1_Bh(v), Mor09e, Mor22e, E1u, R7v+, and VP as the most influential inputs were correlated with the CAD response by virtue of ANN applying a backpropagation learning rule. External validation on previously unseen substances showed that the developed 13-6-3-1 ANN model could be used for CAD response prediction across the examined experimental domain reliably (R2 0.989 and RMSE 0.036). The obtained network was used to reveal CAD response correlations. The impact of organic modifier content and flow rate was in accordance with the theory of the detector's functioning. Additionally, the significance of SpMin1_Bh(v) aided in emphasizing the often neglected surface-dependent CAD character, while the importance of Mor22e as a molecular descriptor accentuated its dependency on the number of electronegative atoms taking part in charging the formed particles. The significance of PFPA demonstrated the possibility of using evaporative chaotropic reagents in CAD response improvement when dealing with highly polar substances that act as kosmotropes. The network was also used in identifying possible interactions between the most significant inputs. A joint effect of PFPA and acetone was shown, representing a good starting point for further investigation with different and, especially, eco-friendly organic solvents and chaotropic agents in the routine application of CAD.

Resistive Memory-Based Analog Synapse: The Pursuit for Linear and Symmetric Weight Update

This article reviews the recent developments in a type of random access memory (RAM) called resistive RAM (RRAM) for the analog synapse, which is an important building block for neuromorphic computing systems. To achieve high learning accuracy in an artificial neural network based on the backpropagation learning rule, a linear and symmetric weight update behavior of the analog synapse is critical. The physical mechanisms in the RRA M (interfacing switching versus filamentary switching) are discussed, and the pros and cons of each mechanism to emulate the analog synaptic weights are compared. Then, various strategies from a materials and device engineering perspective are surveyed to achieve linearly and symmetric conductance changes under identical pulses. Finally, future research directions are outlined.

Multilayer feed forward neural networks for non-linear continuous bidirectional associative memory

The study of Bidirectional associative memory (BAM), with recurrent neural networks and symmetric as well as asymmetric weights, has already been undertaken in various different ways. Using two phases of learning for multilayer neural network architecture in the present paper, a multilayer feed forward neural network model has been proposed to construct the non-linear continuous BAM for pattern association. In the first phase an input pattern is presented to input layer and back propagation learning rule is used to train the network in the feed forward direction for the corresponding associated output pattern. In second phase the output pattern is presented to output layer as input and again the back propagation learning rule is used to train the same network in feedback direction for the corresponding associated input pattern. In these two passes i.e. forward pass and backward pass, the interconnection weights are considered asymmetric. This training process continues till the network does not converge to the final optimal weights by minimizing the mean square errors in both the directions simultaneously. At this convergence of weights the input and output layers exhibit the stability and the performance of such type of BAM is evaluated for the test pattern set while the simulation results exhibit the better performance of associative memory for the proposed method. The storage capacity of network is also increased due to the non-linear mapping between input-output pattern pairs and also the occurrence of spurious states reduces during the recalling process.

Synaptic Characteristics of Ag/AgInSbTe/Ta-Based Memristor for Pattern Recognition Applications

The memristor, a promising candidate for synaptic interconnections in artificial neural network, has gained significant attention for application to neuromorphic systems. One common method is using two memristors as one synapse to represent the positive and negative weights. In this paper, the synaptic behavior of a Ag/AgInSbTe/Ta (AIST)-based memristor is experimentally demonstrated. In addition, a neural architecture using one AIST memristor as a synapse is proposed, where both the plus and minus weights of the neural synapses are realized in a single memristive array. Moreover, the memristor-based neural network is extended to a multilayer architecture, and modified memristor-based backpropagation learning rules are implemented on-chip to achieve pattern recognition. The effects of device variations and input noise on the performance of a memristor-based multilayer neural network (MNN) are also described. The proposed MNN is capable of pattern recognition with high success rates and exhibits several advantages, such as good accuracy, high robustness, and noise immunity.

Hyperbolic Gradient Operator and Hyperbolic Back-Propagation Learning Algorithms.

In this paper, we first extend the Wirtinger derivative which is defined for complex functions to hyperbolic functions, and derive the hyperbolic gradient operator yielding the steepest descent direction by using it. Next, we derive the hyperbolic backpropagation learning algorithms for some multilayered hyperbolic neural networks (NNs) using the hyperbolic gradient operator. It is shown that the use of the Wirtinger derivative reduces the effort necessary for the derivation of the learning algorithms by half, simplifies the representation of the learning algorithms, and makes their computer programs easier to code. In addition, we discuss the differences between the derived Hyperbolic-BP rules and the complex-valued backpropagation learning rule (Complex-BP). Finally, we make some experiments with the derived learning algorithms. As a result, we find that the convergence rates of the Hyperbolic-BP learning algorithms are high even if the fully activation functions are used, and discover that the Hyperbolic-BP learning algorithm for the hyperbolic NN with the split-type hyperbolic activation function has an ability to learn hyperbolic rotation as its inherent property.

An approximate backpropagation learning rule for memristor based neural networks using synaptic plasticity

We describe an approximation to backpropagation algorithm for training deep neural networks, which is designed to work with synapses implemented with memristors. The key idea is to represent the values of both the input signal and the backpropagated delta value with a series of pulses that trigger multiple positive or negative updates of the synaptic weight, and to use the min operation instead of the product of the two signals. In computational simulations, we show that the proposed approximation to backpropagation is well converged and may be suitable for memristor implementations of multilayer neural networks.

Adaptive semi-parallel position/force-sensorless control of electro-hydraulic actuator system using MR fluid damper

The focus of this paper is to develop a semi-parallel control method using an inversion of identification model of a magnetorheological (MR) fluid damper along with a smart predictor controller (SPC) for a damping system using that damper and an electrohydraulic actuator (EHA) in order to realize the real time position/force control of the industrial task requiring interaction with the environment. The inverse model of MR fluid damper is established base on a self-tuning Lyapunov-based fuzzy (STLF) model. This STLF model is designed in the form of a center average fuzzy interference system, of which the fuzzy rules are planted based on the Lyapunov stability condition. In addition, in order to optimize the STLF model, the back propagation learning rules are used to adjust the fuzzy weighting net. Meanwhile, the SPC is constructed using a nonlinear PID controller (NPID) base on feedforward neural network and a smart Grey-Markov predictor (SGMP). Here, the NPID controller is built to drive the system to desired targets. Additionally, a learning mechanism with robust checking conditions is implemented into the NPID in order to optimize online its parameters with respect to the control error minimization. Besides, the SGMP with self-tuning ability of the predictor step size takes part in, first, estimating the system.

Classification of a Driver's cognitive workload levels using artificial neural network on ECG signals

An artificial neural network (ANN) model was developed in the present study to classify the level of a driver's cognitive workload based on electrocardiography (ECG). ECG signals were measured on 15 male participants while they performed a simulated driving task as a primary task with/without an N-back task as a secondary task. Three time-domain ECG measures (mean inter-beat interval (IBI), standard deviation of IBIs, and root mean squared difference of adjacent IBIs) and three frequencydomain ECG measures (power in low frequency, power in high frequency, and ratio of power in low and high frequencies) were calculated. To compensate for individual differences in heart response during the driving tasks, a three-step data processing procedure was performed to ECG signals of each participant: (1) selection of two most sensitive ECG measures, (2) definition of three (low, medium, and high) cognitive workload levels, and (3) normalization of the selected ECG measures. An ANN model was constructed using a feed-forward network and scaled conjugate gradient as a back-propagation learning rule. The accuracy of the ANN classification model was found satisfactory for learning data (95%) and testing data (82%).

Neural computation based vector controlled asynchronous motor fed by three levels NPC

This study presents an improved direct torque control based on artificial neural network techniques fed by a three levels neutral-point-clamped inverter for high power asynchronous motor drive. Indeed, the ANN is divided into four sub-networks, which are individually trained: flux estimation (supervised) with dynamic neurons, torque calculation (fixed-weight) with square neurons, flux angle encoder and magnitude calculation (supervised and fixed-weight) with “logsig” neurons and “tansig” neurons. The back-propagation learning rule is used to design the supervised neural network. The simple structure network facilitates a short training and processing times. The validity of the proposed approaches is confirmed by the simulation.

Artificial neural network for modeling nitrate pollution of groundwater in marginal area of Zayandeh-rood River, Isfahan, Iran

Excessive use of chemical fertilizers, especially nitrogen fertilizers to increase crop and improper purification, and delivery of municipal and industrial wastewater are proposed as factors that increase the amount of nitrate in groundwater in this area. Thus, investigation of nitrate contamination as one of the most important environmental problems in groundwater is necessary. In the present study, modeling and estimation of nitrate pollution in groundwater of marginal area of Zayandeh-rood River, Isfahan, Iran, was investigated using water quality and artificial neural networks. 100 wells (77 agriculture well, 13 drinking well and 10 gardens well) in the marginal area of Zayandeh-rood River, Isfahan, Iran were selected. MATLAB software and three-layer Perceptron network were used. The back-propagation learning rule and sigmoid activation function were applied for the training process. After frequent experiments, a network with one hidden layer and 19 neurons make the least error in the process of network training, testing and validation. ANN models can be applied for the investigation of water quality parameters.

A NEW TECHNIQUE TO PREDICT THE FRACTURES DIP USING ARTIFICIAL NEURAL NETWORKS AND IMAGE LOGS DATA

Fractures provide the place for oil and gas to be reserved and they also can provide the pathway for them to move into the well, so having a proper knowledge of them is essential and every year the companies try to improve the existed softwares in this technology. In this work, the new technique is introduced to be added as a new application to the existed softwares such as Petrel and geoframe softwares. The data used in this work are image logs and the other geological logs data of tree wells located in Gachsaran field, wells number GS-A, GS-B and GS-C. The new technique by using the feed-forward artificial neural networks (ANN) with back-propagation learning rule can predict the fracture dip data of the third well using the data from the other 2 wells. The result obtained showed that the ANN model can simulate the relationship between fractures dips in these 3 wells which the multiple R of training and test sets for the ANN model is 0.95099 and 0.912197, respectively.

INVERSION OF THREE LAYERS MULTI-SCALE SPM MODEL BASED ON NEURAL NETWORK TECHNIQUE FOR THE RETRIEVAL OF SOIL MULTI-SCALE ROUGHNESS AND MOISTURE PARAMETERS

Abstract. In this paper, a multi-layered multi-scale backscattering model for a lossy medium and a neural network inversion procedure has been presented. We have used a bi-dimensional multi-scale (2D MLS) roughness description where the surface is considered as a superposition of a finite number of one-dimensional Gaussian processes each one having a spatial scale using the wavelet transform and the Mallat algorithm to describe natural surface roughness. An adapted three layers 2D MLS small perturbations (SPM) model has been used to describe radar backscattering response of semiarid sub-surfaces. The total reflection coefficients of the natural soil are computed using the multilayer model, and volumetric scattering is approximated by the internal reflections between layers. The original multi-scale SPM model includes only the surface scattering of the natural bare soil, while the multilayer soil modified 2D MLS SPM model includes both the surface scattering and the volumetric scattering within the soil. This multi-layered model has been used to calculate the total surface reflection coefficients of a natural soil surface for both horizontal and vertical co-polarizations. A parametric analysis presents the dependence of the backscattering coefficient on multi scale roughness and soil. The overall objective of this work is to retrieve soil surfaces parameters namely roughness and soil moisture related to the dielectric constant by inverting the radar backscattered signal from natural soil surfaces. To perform the inversion of the modified three layers 2D MLS SPM model, we used a multilayer neural network (NN) architecture trained by a back-propagation learning rule.

Hysteresis modeling of magneto-rheological damper using self-tuning Lyapunov-based fuzzy approach

Magneto-rheological (MR) fluid damper is a semi-active control device that has recently received more attention because they offer the adaptability of active control devices without requiring the associated large power sources. But inherent nonlinear nature of the MR fluid damper is one of the challenging aspects for utilizing this device to achieve the high performance. So development of an accurate MR fluid damper model is necessary to take the advantages from its unique characteristics. The focus of this paper is to develop an alternative method for modeling a MR fluid damper by using a so-called self-tuning Lyapunov-based fuzzy model (STLFM). Here, the model is constructed in the form of a center average fuzzy interference system, of which the fuzzy rules are designed based on the Lyapunov stability condition. In addition, in order to optimize the STLFM, the back propagation learning rules are used to adjust the fuzzy weighting net. Firstly, experimental data of a damping system using this damper is used to optimize the model. Next, the optimized model is used to estimate online the damping performance in the real-time conditions. The modeling results prove convincingly that the developed model could represent satisfactorily the behavior of the MR fluid damper.

Hybrid Artificial Intelligent Approach for Determination of Water Saturation using Archie’s Formula in Carbonate Reservoirs

The challenge to determine the accurate water saturation is still faced Petroleum Engineering. The difficulty of this problem will be increase if we deal with carbonate rocks. There are some available techniques used to determine the water saturation. However, the accuracy of those techniques has become unable to find the best results. Several available techniques have been used to estimate water saturation such as conventional, CAPE (a, m, n), CAPE (1, m, n) and 3D methods. Currently, the achievements of Artificial Intelligent (AI) techniques alone open the door to use the hybrid system such as (PSONN). In this model, the Particle Swarm Optimization (PSO) technique is employed to search for optimal connection weighs and thresholds for the neural networks (NN), then the back-propagation learning rule and training algorithm is used to adjust the final weights. A total of about 383 data points obtained from the laboratory measurements of electrical properties from carbonate core plugs of Middle East reservoir were used for the implementation of the proposed technique. Statistical analysis and comparative study show that the performance of PSONN model is the best one with lower root mean square error (0.092) and higher accuracy of correlation coefficient (0.95) than those obtained with previous methods. Results showed that the new hybrid PSONN model outperforms some available methods and overcome the weakness if we use AI alone. From error analysis, it is found that CAPE and 3-D and PSONN methods ensure minimum error of water saturation values.