Multilayer Recurrent Network Research Articles

A vector-to-sequence based multilayer recurrent network surrogate model for history matching of large-scale reservoir

History matching can estimate the parameter of spatially varying geological properties and provide reliable numerical models for reservoir development and management. However, in practice, high-dimension, multiple-solutions and computational cost are key issues that restrict the application of history matching methods. Recently, the combination of deep-learning-based surrogate model and sampling algorithm has been widely studied in history matching to overcome the limitations. Considering that real-world large-scale reservoirs often have hundreds of thousands or even millions of grid-based uncertain parameters, extracting spatial features using convolutional neural networks requires a lot of computational cost and storage requirements. Therefore, in this work, we mainly study how to use the recurrent neural network (RNN) to construct the surrogate model for history matching. Specifically, we propose a multilayer RNN surrogate model based on a vector-to-sequence modeling framework. The multilayer RNN surrogate model with gated recurrent unit (GRU), termed MLGRU, is developed to approximate the mapping from feature vector of geological realizations to the production data. The feature vector is the low-dimensional representation of geological parameter fields after using the re-parameterization method, while production data are the simulation results of historical period. In addition, we design a log-transformation-based windowed normalization (LTWN) method for the production data, which can enhance the learnability and features of production data. The MLGRU model is incorporated into a multimodal estimation of distribution algorithm (MEDA) to formulate a history matching workflow. The hyperparameters and performance of the proposed MLGRU model are analyzed by numerical experiments on a 2D reservoir model. Furthermore, numerical experiments performed on the Brugge benchmark model, a large-scale 3D reservoir model, demonstrated the performance of the proposed surrogate model and history matching method.

Locally recurrent neural networks for long-term wind speed and power prediction

The paper deals with a real-world application, the long-term wind speed and power forecasting in a wind farm using locally recurrent multilayer networks as forecast models. To cope with the complexity of the process and to improve the performance of the models, a class of optimal on-line learning algorithms is employed for training the locally recurrent networks based on the recursive prediction error (RPE) algorithm. A global RPE algorithm is devised and three local learning algorithms are suggested by partitioning the GRPE into a set of sub-problems at the neuron level to reduce computational complexity and storage requirements. Experimental results on the wind prediction problem demonstrate that the proposed algorithms exhibit enhanced performance, in terms of convergence speed and the accuracy of the attained solutions, compared to conventional gradient-based methods. Furthermore, it is shown that the suggested recurrent forecast models outperform the atmospheric and time-series models.

Training trajectories by continuous recurrent multilayer networks

This paper addresses the problem of training trajectories by means of continuous recurrent neural networks whose feedforward parts are multilayer perceptrons. Such networks can approximate a general nonlinear dynamic system with arbitrary accuracy. The learning process is transformed into an optimal control framework where the weights are the controls to be determined. A training algorithm based upon a variational formulation of Pontryagin's maximum principle is proposed for such networks. Computer examples demonstrating the efficiency of the given approach are also presented.

Forecasting nitrogen dioxide concentration in ambient air using artificial Neural‐networks

Artificial Neural‐networks is employed to predict the nitrogen dioxide concentration during November, 1997 to February, 1998 at three sites each representing industrial, commercial and residential activity respectively in Mumbai. The application of the Multilayer Recurrent network with back‐propagation learning algorithm is reported in the prediction at three sites using the meteorological variables at one site. The generalization ability of the model is confirmed by root mean square error and correlation between observed and predicted concentrations. The evaluation of model results shows that the degree of success in forecasting NO2 concentration is promising.

Physiology and morphology of olfactory neurons associating with the protocerebral lobe of the honeybee brain

Physiology and morphology of olfactory neurons associated with the protocerebral lobe around the α-lobe of the mushroom body were studied in the brain of the honeybee Apis mellifera using intracellular recording and staining techniques. The responses of neurons to behaviorally relevant odorants (a blend, and components of the Nasonov pheromone, and some other non-pheromonal odors) were recorded. Different response patterns were observed within different neurons, and often within the same neuron, in response to different stimuli. All the neurons stained had innervations in the protocerebral lobe. The cell profiles varied from cells connecting the antennal lobe with both the protocerebral and lateral protocerebral lobes (projection neurons), cells linking the pedunculus of the mushroom body with both the protocerebral and lateral protocerebral lobes (PE1 neurons), cells linking the α-lobe and protocerebral lobe with the calyces of the mushroom body (feedback neurons), and cells linking the α-lobe and protocerebral lobe with the antennal lobe (recurrent neurons), to cells connecting the protocerebral lobe with the contralateral protocerebrum (bilateral neurons). These findings suggest that the protocerebral lobe acts as an olfactory center associating with other centers, and provides multi-layered recurrent networks within the protocerebrum and between the deutocerebrum and the protocerebrum in honeybee olfactory pathways.

Neural networks for solid transport modelling in sewer systems during storm events

Models for solid transport in sewers during storm events are increasingly used by engineers and operators to improve their systems and the quality of receiving waters. However, a major difficulty that prevents more general use of these models is their calibration, which requires field data, accurate information about catchments and sewers, and a specific methodology. Therefore, research has been carried out to assess the ability of connectionist models to reproduce and replace usual models for use by an operator. Such models require fewer data, are self-calibrated, and very easy to use. The first stage presented in this paper consists in a comparison between neural networks and the HYPOCRAS model, using simulations of real pollutographs for single storm events. Two specific recurrent neural networks based on the HYPOCRAS model and a general-purpose recurrent multilayer network are used to simulate hydrographs and pollutographs of TSS. The learning algorithm and the performance criterion used for optimization of these networks are described in detail. Experimental results with simulated and real data are then presented.

Adaptive robot control using neural networks

This paper studies the trajectory tracking problem to control the nonlinear dynamic model of a robot using neural networks. These controllers are based on learning from input-output measurements and not on parametric-model-based dynamics. Multilayer recurrent networks are used to estimate the dynamics of the system and the inverse dynamic model. The training is achieved using the backpropagation method. The minimization of the quadratic error is computed by a variable step gradient method. Another multilayer recurrent neural network is added to estimate the joint accelerations. The control process is applied to a two degree-of-freedom (DOF) SCARA robot using a DSP-based controller. Experimental results show the effectiveness of this approach. The tracking trajectory errors are very small and torques expected at manipulator joints are free of chattering. >

Weight perturbation: an optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks

Previous work on analog VLSI implementation of multilayer perceptrons with on-chip learning has mainly targeted the implementation of algorithms such as back-propagation. Although back-propagation is efficient, its implementation in analog VLSI requires excessive computational hardware. It is shown that using gradient descent with direct approximation of the gradient instead of back-propagation is more economical for parallel analog implementations. It is shown that this technique (which is called ;weight perturbation') is suitable for multilayer recurrent networks as well. A discrete level analog implementation showing the training of an XOR network as an example is presented.

Weight Perturbation: An Optimal Architecture and Learning Technique for Analog VLSI Feedforward and Recurrent Multilayer Networks.

Previous work on analog VLSI implementation of multilayer perceptrons with on-chip learning has mainly targeted the implementation of algorithms like backpropagation. Although backpropagation is efficient, its implementation in analog VLSI requires excessive computational hardware. In this paper we show that, for analog parallel implementations, the use of gradient descent with direct approximation of the gradient using "weight perturbation" instead of backpropagation significantly reduces hardware complexity. Gradient descent by weight perturbation eliminates the need for derivative and bidirectional circuits for on-chip learning, and access to the output states of neurons in hidden layers for off-chip learning. We also show that weight perturbation can be used to implement recurrent networks. A discrete level analog implementation showing the training of an XOR network as an example is described.