Gradient Descent Training Research Articles

Memristor-based multilayer neural networks with online gradient descent training.

Learning in multilayer neural networks (MNNs) relies on continuous updating of large matrices of synaptic weights by local rules. Such locality can be exploited for massive parallelism when implementing MNNs in hardware. However, these update rules require a multiply and accumulate operation for each synaptic weight, which is challenging to implement compactly using CMOS. In this paper, a method for performing these update operations simultaneously (incremental outer products) using memristor-based arrays is proposed. The method is based on the fact that, approximately, given a voltage pulse, the conductivity of a memristor will increment proportionally to the pulse duration multiplied by the pulse magnitude if the increment is sufficiently small. The proposed method uses a synaptic circuit composed of a small number of components per synapse: one memristor and two CMOS transistors. This circuit is expected to consume between 2% and 8% of the area and static power of previous CMOS-only hardware alternatives. Such a circuit can compactly implement hardware MNNs trainable by scalable algorithms based on online gradient descent (e.g., backpropagation). The utility and robustness of the proposed memristor-based circuit are demonstrated on standard supervised learning tasks.

Nonlinear system identification using a cuckoo search optimized adaptive Hammerstein model

An attempt has been made in this paper to model a nonlinear system using a Hammerstein model. The Hammerstein model considered in this paper is a functional link artificial neural network (FLANN) in cascade with an adaptive infinite impulse response (IIR) filter. In order to avoid local optima issues caused by conventional gradient descent training strategies, the model has been trained using a cuckoo search algorithm (CSA), which is a recently proposed stochastic algorithm. Modeling accuracy of the proposed scheme has been compared with that obtained using other popular evolutionary computing algorithms for the Hammerstein model. Enhanced modeling capability of the CSA based scheme is evident from the simulation results.

Improving the Performance of Intelligent Back Analysis for Tunneling Using Optimized Fuzzy Systems: Case Study of the Karaj Subway Line 2 in Iran

Tunnels are often designed by uncertain geotechnical data. In order to reduce these uncertainties, back analysis is commonly selected to re-estimate the assumed parameters. This paper presents a novel, intelligent back analysis method combining fuzzy systems, imperialistic competitive algorithm, and numerical analysis. The proposed methodology comprises three phases. First, a database of a real case study and numerical analysis are used to develop the training and testing data of the study. In the second phase, the nonlinear relationship of two sets of parameters, including geomechanical parameters of the soil mass and the zone stress conditions, with surface settlement is investigated by three fuzzy models. These models are designed by three methods including particle swarm optimization, imperialistic competitive algorithm, and integration of nearest neighborhood clustering with gradient descent training. In the last phase, imperialistic competitive algorithm is employed one more time to implement the back analysis procedure in the three tuned fuzzy models. Finally, verification of the models is done with the numerical analysis on the results of back analysis, and then the results are compared with the measured values of settlements. The results introduced the particle swarm optimization tuned fuzzy model as the most accurate intelligent model.

Training neural networks with ant colony optimization algorithms for pattern classification

Feed-forward neural networks are commonly used for pattern classification. The classification accuracy of feed-forward neural networks depends on the configuration selected and the training process. Once the architecture of the network is decided, training algorithms, usually gradient descent techniques, are used to determine the connection weights of the feed-forward neural network. However, gradient descent techniques often get trapped in local optima of the search landscape. To address this issue, an ant colony optimization (ACO) algorithm is applied to train feed-forward neural networks for pattern classification in this paper. In addition, the ACO training algorithm is hybridized with gradient descent training. Both standalone and hybrid ACO training algorithms are evaluated on several benchmark pattern classification problems, and compared with other swarm intelligence, evolutionary and traditional training algorithms. The experimental results show the efficiency of the proposed ACO training algorithms for feed-forward neural networks for pattern classification.

A Comparative Study of MLP Networks Using Backpropagation Algorithms in Electrical Equipment Thermography

Nowadays, infrared thermographic inspection has increasingly been utilized for fault diagnosis tools of electrical equipment, because it maintains non-interrupting operation of power system and ensures early diagnosis of faults. The article discusses the performance comparison of multilayer perceptron (MLP) networks using various backpropagation algorithms for thermal imaging-based condition monitoring of electrical hotspots. The training algorithms are Levenberg–Marquardt, Broyden–Fletcher–Goldfarb–Shanno quasi-Newton, resilient backpropagation, gradient descent with momentum and adaptive learning rate and standard gradient descent training algorithms. The performance of the training algorithms are evaluated in terms of percentage of accuracy, sensitivity, specificity, false-positive and false-negative results. In the beginning, thermal images of electrical equipment are captured using infrared camera. Six statistical intensity features, namely mean, maximum, minimum, median, standard deviation and variance extracted from each hotspot of equipment thermal image are used as the inputs of multilayered perceptron networks to classify the thermal conditions of hotspots into two classes, namely normal and defective. The comparison of MLP networks using training algorithms proves that the MLP network trained with resilient backpropagation algorithm gives the best performance in thermal condition recognition.

Time series forecasting using a deep belief network with restricted Boltzmann machines

Multi-layer perceptron (MLP) and other artificial neural networks (ANNs) have been widely applied to time series forecasting since 1980s. However, for some problems such as initialization and local optima existing in applications, the improvement of ANNs is, and still will be the most interesting study for not only time series forecasting but also other intelligent computing fields. In this study, we propose a method for time series prediction using Hinton and Salakhutdinov׳s deep belief nets (DBN) which are probabilistic generative neural network composed by multiple layers of restricted Boltzmann machine (RBM). We use a 3-layer deep network of RBMs to capture the feature of input space of time series data, and after pretraining of RBMs using their energy functions, gradient descent training, i.e., back-propagation learning algorithm is used for fine-tuning connection weights between “visible layers” and “hidden layers” of RBMs. To decide the sizes of neural networks and the learning rates, Kennedy and Eberhart׳s particle swarm optimization (PSO) is adopted during the training processes. Furthermore, “trend removal”, a preprocessing to the original data, is also approached in the forecasting experiment using CATS benchmark data. Additionally, approximating and short-term prediction of chaotic time series such as Lorenz chaos and logistic map were also applied by the proposed method.

Fracture density estimation from core and conventional well logs data using artificial neural networks: The Cambro-Ordovician reservoir of Mesdar oil field, Algeria

Fracture density estimation is an indisputable challenge in fractured reservoir characterization. Traditional techniques of fracture characterization from core data are costly, time consuming, and difficult to use for any extrapolation to non-cored wells. The aim of this paper is to construct a model able to predict fracture density from conventional well logs calibrated to core data by using artificial neural networks (ANNs). This technique was tested in the Cambro-Ordovician clastic reservoir from Mesdar oil field (Saharan platform, Algeria). For this purpose, 170 cores (2120.14m) from 17 unoriented wells have been studied in detail. Seven training algorithms and eight neuronal network architectures were tested.The best architecture is a four layered [6-16-3-1] network model with: a six-neuron input layer (Gamma ray, Sonic interval transit time, Caliper, Neutron porosity, Bulk density logs and core depth), two hidden layers; the first hidden layer has 16 neurons, the second one has three neurons. And a one-neuron output layer (fracture density). The results based on 8094 data points from 13 wells show the excellent prediction ability of the conjugate gradient descent (CGD) training algorithm (R-squared=0.812).The cross plot of measured and predicted values of fracture density shows a very high coefficient of determination of 0.848. Our studies have demonstrated a good agreement between our neural network model prediction and core fracture measurements. The results are promising and can be easily extended in other similar neighboring naturally fractured reservoirs.

River suspended sediment estimation by climatic variables implication: Comparative study among soft computing techniques

Estimating sediment volume carried by a river is an important issue in water resources engineering. This paper compares the accuracy of three different soft computing methods, Artificial Neural Networks (ANNs), Adaptive Neuro-Fuzzy Inference System (ANFIS), and Gene Expression Programming (GEP), in estimating daily suspended sediment concentration on rivers by using hydro-meteorological data. The daily rainfall, streamflow and suspended sediment concentration data from Eel River near Dos Rios, at California, USA are used as a case study. The comparison results indicate that the GEP model performs better than the other models in daily suspended sediment concentration estimation for the particular data sets used in this study. Levenberg–Marquardt, conjugate gradient and gradient descent training algorithms were used for the ANN models. Out of three algorithms, the Conjugate gradient algorithm was found to be better than the others.

Neuro-fuzzy integrated system with its different domain applications

Computational intelligence combines fuzzy systems, neural network and evolutionary computing. In this paper, architecture of a neuro–fuzzy integrated system is presented. A new kind of error backpropagation algorithm to adjust the membership functions of each variable and optimise fuzzy rules is developed. To minimise the output error, a variational method for determining globally optimal learning parameters and learning rules for online gradient descent training of multilayer neural network has been proposed. In order to show the effectiveness of the proposed system, simulation for different variety of domain has been performed. The controller for inverted pendulum has been demonstrated. The controller uses error backpropagation algorithm to adjust the membership functions of each variable, optimise fuzzy rules, and identify the inverted pendulum. Neuro–fuzzy integrated system for coronary heart disease has also been simulated. The results suggest that this kind of hybrid system is also suitable for the identification of patients with high/low cardiac risk.

Active fin control for ship stabilization system using heuristic genetic optimization

Active fin control is the most effective anti-rolling approach to ship stabilization. However, an accurate model of the whole nonlinear dynamic ship system under random wave or wind impact is difficult to obtain. This study develops an intelligent guarded heuristic genetic algorithm fin controller, including a heuristic genetic algorithm fin controller and a guarded fin controller, to identify the optimum solution for ship stabilization system. In the heuristic genetic algorithm fin controller design, the gradient-descent training is embedded into a traditional genetic algorithm to construct the main controller, which consequently determines the optimum fin control angle in response to uncertainties. To ensure that the system states are around a defined bound region, the proposed system uses a guarded fin controller to adjust the control angle. The stabilization system uses a gyroscope and accelerometer to detect rolling conditions, and the gathered data are fed to an embedded microcontroller to calculate output commands. To verify the performance of the proposed guarded heuristic genetic algorithm fin controller, an analogous two-wheeled robot balance model and experimental platform are adopted in the simulations and experiments as the preliminary tool. Simulations and experimental results confirm the effectiveness of the proposed system, and this study compares its performance with other fin control schemes.

Recurrent neural tracking control based on multivariable robust adaptive gradient-descent training algorithm

In this paper, a recurrent neural network (RNN) based robust tracking controller is designed for a class of multiple-input-multiple-output discrete time nonlinear systems. The RNN is used in the closed-loop system to estimate online unknown nonlinear system function. A multivariable robust adaptive gradient-descent training algorithm is developed to train RNN. The weight convergence and system stability are proven in the sense of Lyapunov function. Simulation results are presented for a two-link robot tracking control problem.

Design of GA-Based Control for Electrical Servo Drive

An alternative control scheme including a directional genetic algorithm controller (DGAC) and a supervisory controller is developed to control the position of an electrical servo drive in this study. In the DGAC design, the spirit of gradient descent training is embedded in genetic algorithm (GA) to construct a main controller to search optimum control effort under possible occurrence of uncertainties. In order to ensure the system states around a defined bound region, a supervisory controller, which is derived in the sense of Lyapunov stability theorem, is added to adjust the control effort. Compared with enunciated GA control methods, the proposed control scheme possesses some salient advantages of simple framework, fewer executing time and good self-organizing properties even for nonlinear dynamical system. The effectiveness is demonstrated by simulation results, and its advantages are indicated in comparison with other GA control schemes for a field-oriented control induction motor drive.

Adaptive wavelet neural network-based fast dynamic available transfer capability determination

An adaptive wavelet neural network (AWNN)-based method has been proposed to determine dynamic available transfer capability (DATC) in the electricity markets, having bilateral as well as multilateral contracts. Mexican hat wavelet basis function has been used as the activation function in the hidden layer of the network. Wavelet parameters, that is, translations and dilations of the AWNN, have been initialised using Euclidean distance-based clustering method. The AWNN has been trained using back propagation gradient descent training algorithm. The relevant features to be used, as input to the AWNN, are identified using a random forest technique. To demonstrate the effectiveness of the proposed AWNN-based method for the DATC determination, it has been tested on 39-bus New England system and a 246-bus Indian system and its results have been compared to the radial basis function neural network (RBFNN).

An approach for the aging process optimization of Al–Zn–Mg–Cu series alloys

A new model based on least square support vector machines (LSSVM) and capable of forecasting mechanical and electrical properties of Al–Zn–Mg–Cu series alloys has been proposed for the first time. Data mining and artificial intelligence techniques of aluminum alloys are used to examine the forecasting capability of the model. In order to improve predictive accuracy and generalization ability of LSSVM model, a grid algorithm and cross-validation technique has been adopted to determine the optimal hyper-parameters of LSSVM automatically. The forecasting performance of the LSSVM model and the artificial neural network (ANN) has been compared with the experimental values. The result shows that the LSSVM model provides slightly better capability of generalized prediction compared to back propagation network (BPN) in combination with the gradient descent training algorithm. Considering its advantages of the computation speed, unique optimal solution, and generalization performance, the LSSVM model is therefore considered to be used as an alternative powerful modeling tool for the aging process optimization of aluminum alloys. Furthermore, a novel methodology hybridizing nondominated sorting-based multi-objective genetic algorithm (MOGA) and LSSVM has been proposed to make tradeoffs between the mechanical and electrical properties. A desirable nondominated solution set has been obtained and reported.

Mineral Potential Mapping Using Bayesian Learning for Multilayer Perceptrons

Multilayer perceptrons (MLPs) can be used to discover a function which can be used to map from a set of input variables onto a value representing the conditional probability of mineralization. The standard approach to training MLPs is gradient descent, in which the error between the network output and the target output is reduced in each iteration of the training algorithm. In order to prevent overfitting, a split-sample validation procedure is used, in which the data is partitioned into two sets: a training set, which is used for weight optimization, and a validation set, which is used to optimize various parameters that can be used to prevent overfitting. One of the problems with this approach is that the resulting maps can display significant variability which stems from (i) the (randomly initialized) starting weights and (ii) the particular training/validation set partition (also determined randomly). This problem is especially pertinent on mineral potential mapping tasks, in which the number of deposit cells is a very small proportion of the total number of cells in the study area. In contrast to gradient descent methods, Bayesian learning techniques do not find a single weight vector; rather, they infer the posterior distribution of the weights given the data. Predictions are then made by integrating over this distribution. An important advantage of the Bayesian approach is that the optimization of parameters such as the weight decay regularization coefficient can be performed using training data alone, thus avoiding the noise introduced through split-sample validation. This paper reports results of applying Bayesian learning techniques to the production of maps representing gold mineralization potential over the Castlemaine region of Victoria, Australia. Maps produced using the Bayesian approach display significantly less variability than those produced using gradient descent training. They are also more reliable at predicting the presence of unknown deposits.

A block bigram prediction model for statistical machine translation

In this article, we present a novel training method for a localized phrase-based prediction model for statistical machine translation (SMT). The model predicts block neighbors to carry out a phrase-based translation that explicitly handles local phrase reordering. We use a maximum likelihood criterion to train a log-linear block bigram model which uses real-valued features (e.g., a language model score) as well as binary features based on the block identities themselves (e.g., block bigram features). The model training relies on an efficient enumeration of local block neighbors in parallel training data. A novel stochastic gradient descent (SGD) training algorithm is presented that can easily handle millions of features. Moreover, when viewing SMT as a block generation process, it becomes quite similar to sequential natural language annotation problems such as part-of-speech tagging, phrase chunking, or shallow parsing. Our novel approach is successfully tested on a standard Arabic-English translation task using two different phrase reordering models: a block orientation model and a phrase-distortion model.

Artificial neural networks based on principal component analysis input selection for clinical pattern recognition analysis

Two clinical data sets were applied for pattern recognition in order to discover the correlation between urinary nucleoside profiles and tumours. One data set contains 168 clinical urinary samples, of which 84 specimens are from female thyroid cancer patients (malignant tumour group), and the other samples were collected from healthy women (normal group). However, 168 clinical urinary samples comprised the second data set, too. In all the specimens, each number of the samples for both uterine cervical cancer patients (malignant tumour group) and healthy females (normal group) is 60, and the other 48 samples were collected from uterine myoma patients (benign tumour group). For the two data sets, the separation and quantitative determination of the clinical urinary nucleosides were performed by capillary electrophoresis (CE). The pattern recognition was achieved applying multiple layer perceptron artificial neural networks (MLP ANN) based on conjugate gradient descent training algorithm. Moreover, applying the proposed principal component analysis (PCA) input selection scheme to MLP ANN, the accuracy rate of the pattern recognition was improved to some extent (or without any deterioration) even by much simpler structure of MLP ANN. The study showed that MLP ANN based on PCA input selection was a promising tool for pattern recognition.

Content-based image retrieval using fuzzy perceptual feedback

In this paper, a new framework called fuzzy relevance feedback in interactive content-based image retrieval (CBIR) systems is introduced. Conventional binary labeling scheme in relevance feedback requires a crisp decision to be made on the relevance of the retrieved images. However, it is inflexible as user interpretation of visual content varies with respect to different information needs and perceptual subjectivity. In addition, users tend to learn from the retrieval results to further refine their information requests. It is, therefore, inadequate to describe the user's fuzzy perception of image similarity with crisp logic. In view of this, we propose a fuzzy relevance feedback approach which enables the user to make a fuzzy judgement. It integrates the user's fuzzy interpretation of visual content into the notion of relevance feedback. An efficient learning approach is proposed using a fuzzy radial basis function (FRBF) network. The network is constructed based on the user's feedbacks. The underlying network parameters are optimized by adopting a gradient-descent training strategy due to its computational efficiency. Experimental results using a database of 10,000 images demonstrate the effectiveness of the proposed method.

On global-local artificial neural networks for function approximation.

We present a hybrid radial basis function (RBF) sigmoid neural network with a three-step training algorithm that utilizes both global search and gradient descent training. The algorithm used is intended to identify global features of an input-output relationship before adding local detail to the approximating function. It aims to achieve efficient function approximation through the separate identification of aspects of a relationship that are expressed universally from those that vary only within particular regions of the input space. We test the effectiveness of our method using five regression tasks; four use synthetic datasets while the last problem uses real-world data on the wave overtopping of seawalls. It is shown that the hybrid architecture is often superior to architectures containing neurons of a single type in several ways: lower mean square errors are often achievable using fewer hidden neurons and with less need for regularization. Our global-local artificial neural network (GL-ANN) is also seen to compare favorably with both perceptron radial basis net and regression tree derived RBFs. A number of issues concerning the training of GL-ANNs are discussed: the use of regularization, the inclusion of a gradient descent optimization step, the choice of RBF spreads, model selection, and the development of appropriate stopping criteria.

Supervisory control for linear piezoelectric ceramic motor drive using genetic algorithm

This paper presents a supervisory genetic algorithm (SGA) control system for a linear piezoelectric ceramic motor (LPCM) driven by a two-inductance two-capacitance resonant driving circuit. First, the motor configuration and driving circuit of an LPCM are introduced, and its hypothetical dynamic model is described briefly. Since the dynamic characteristics and motor parameters of the LPCM are highly nonlinear and time varying, an SGA control system is therefore investigated to achieve high-precision position control. The proposed SGA control system is composed of two parts. One is a GA control that is utilized to search an optimum control effort online via gradient descent training process, and the other is a supervisory control to stabilize the system states around a predefined bound region. Compared with conventional GA control systems, the proposed control scheme possesses the salient advantages of simple structure, fewer executing time, and good self-organizing properties. The effectiveness of the proposed driving circuit and control system is verified with numerical simulations and hardware experiments under the occurrence of uncertainties. In addition, the advantages of the proposed control scheme are indicated in comparison with a traditional integral-proportional position-control system.