Multilayer Recurrent Neural Network Research Articles

Deep Graph Learning-Based Surrogate Model for Inverse Modeling of Fractured Reservoirs

Inverse modeling can estimate uncertain parameters in subsurface reservoirs and provide reliable numerical models for reservoir development and management. The traditional simulation-based inversion method usually requires numerous numerical simulations, which is time-consuming. Recently, deep learning-based surrogate models have been widely studied as an alternative to numerical simulation, which can significantly improve the solving efficiency of inversion. However, for reservoirs with complex fracture distribution, constructing the surrogate model of numerical simulation presents a significant challenge. In this work, we present a deep graph learning-based surrogate model for inverse modeling of fractured reservoirs. Specifically, the proposed surrogate model integrates the graph attention mechanisms to extract features of fracture network in reservoirs. The graph learning can retain the discrete characteristics and structural information of the fracture network. The extracted features are subsequently integrated with a multi-layer recurrent neural network model to predict the production dynamics of wells. A surrogate-based inverse modeling workflow is then developed by combining the surrogate model with the differential evolutionary algorithm. Numerical studies performed on a synthetic naturally fractured reservoir model with multi-scale fractures illustrate the performance of the proposed methods. The results demonstrate that the proposed surrogate model exhibits promising generalization performance of production prediction. Compared with tens of thousands of numerical simulations required by the simulation-based inverse modeling method, the proposed surrogate-based method only requires 1000 to 1500 numerical simulations, and the solution efficiency can be improved by ten times.

RNN-Based Time Series Analysis for Wind Turbine Energy Forecasting

One significant source of renewable energy is wind power, which has the potential to generate sustainable energy. However, wind turbines have many challenges, such as high initial investment costs, the dynamic nature of wind speed, and the challenge of locating wind-efficient energy regions. Wind power predicting is crucial for effective planning of wind power generation, optimization of power generation, grid integration, and security of supply. Therefore, highly accurate forecasts ensure the efficient and sustainable operation of the wind energy sector and contribute to energy security, economic stability, and environmental sustainability. This study proposes a deep learning (DL) approach based on recurrent neural networks (RNNs) for long-term wind power forecasting utilizing climatic data. The input data that forms the basis of this study is obtained directly from a wind turbine system operating under real-world conditions. The proposed model in this study is based on a multilayer back-propagation neural network (RNN) architecture specifically designed to effectively handle complex data sets and time-dependent series. The architecture of the model is built on an RNN consisting of four separate layers, each with 50 hidden neurons, carefully structured to increase its capacity to capture complex features. To improve the robustness of the model and avoid overlearning, each RNN layer is followed by a dropout (regularizing) layer that randomly deactivates 20% of the neurons to enhance the generalization ability of the network. To finalize the prediction capability of the model, a linear function was chosen in the last layer to directly match the actual values. Evaluating the model performance metrics, the proposed architecture achieved a prediction accuracy of 91% R2 on the test dataset. The findings indicate that proposed method based on multilayer RNN can successfully capture the relationships between the sequential data of the wind turbine.

ES-dRNN: A Hybrid Exponential Smoothing and Dilated Recurrent Neural Network Model for Short-Term Load Forecasting.

Short-term load forecasting (STLF) is challenging due to complex time series (TS) which express three seasonal patterns and a nonlinear trend. This article proposes a novel hybrid hierarchical deep-learning (DL) model that deals with multiple seasonality and produces both point forecasts and predictive intervals (PIs). It combines exponential smoothing (ES) and a recurrent neural network (RNN). ES extracts dynamically the main components of each individual TS and enables on-the-fly deseasonalization, which is particularly useful when operating on a relatively small dataset. A multilayer RNN is equipped with a new type of dilated recurrent cell designed to efficiently model both short and long-term dependencies in TS. To improve the internal TS representation and thus the model's performance, RNN learns simultaneously both the ES parameters and the main mapping function transforming inputs into forecasts. We compare our approach against several baseline methods, including classical statistical methods and machine learning (ML) approaches, on STLF problems for 35 European countries. The empirical study clearly shows that the proposed model has high expressive power to solve nonlinear stochastic forecasting problems with TS including multiple seasonality and significant random fluctuations. In fact, it outperforms both statistical and state-of-the-art ML models in terms of accuracy.

An Explainable and Generalizable Recurrent Neural Network Approach for Differentiating Human Brain States on EEG Dataset.

Electroencephalogram (EEG) is one of the most widely used brain computer interface (BCI) approaches. Despite the success of existing EEG approaches in brain state recognition studies, it is still challenging to differentiate brain states via explainable and generalizable deep learning approaches. In other words, how to explore meaningful and distinguishing features and how to overcome the huge variability and overfitting problem still need to be further studied. To alleviate these challenges, in this work, a multiple random fragment search-based multilayer recurrent neural network (MRFS-MRNN) is proposed to improve the differentiating performance and explore meaningful patterns. Specifically, an explainable MRNN module is proposed to capture the temporal dependences preserved in EEG time series. Besides, a MRFS module is designed to cut multiple random fragments from the entire EEG signal time course to improve the effectiveness of brain state differentiating ability. MRFS-MRNN is concatenatedto effectively overcome the huge variabilities and overfitting problems. Experiment results demonstrate that the proposed MRFS-MRNN model not only has excellent differentiating performance, but also has good explanation and generalization ability. The classification accuracies reach as high as 95.18% for binary classification and 89.19% for four-category classification on the individual level. Similarly, 95.53% and 85.84% classification accuracies are obtained for the binary and four-category classification on the group level. What's more, 94.28% and 85.43% classification accuracies of binary and four-category classifications are achieved for predicting brand new subjects. The experiment results showed that the proposed method outperformed other state-of-the-art (SOTA) models on the same underlying data and improved the explanation and generalization ability.

Estimation of Iron Loss in Permanent Magnet Synchronous Motors Based on Particle Swarm Optimization and a Recurrent Neural Network

The popularity of permanent magnet synchronous motors (PMSMs) has increased in recent years due to their high efficiency, compact size, and low maintenance needs. Calculating iron loss in PMSMs is crucial for designing and optimizing PMSMs to achieve high efficiency and a long lifespan, as this can significantly affect motor performance. However, multiple factors influence the accuracy of iron loss calculations in PMSMs, including the intricate magnetic behavior of the motor under different operating conditions, as well as the influence of the motor’s dynamic behavior during the operation process. This paper proposes a method based on particle swarm optimization (PSO) and a recurrent neural network (RNN) to estimate the iron loss in PMSMs, independent of the empirical iron loss formula. This method establishes an iron loss calculation model considering high-order harmonics, rotating magnetization, and temperature factors. Accounting for the multifactor influence, the model studies the law of loss change under different magnetic flux densities, frequencies, and temperature conditions. To avoid the deviation problem caused by conventional polynomial fitting, a multilayer RNN and PSO are used to train and optimize the neural network. Iron loss in complex cases beyond the measurement range can be accurately estimated. The proposed method helps achieve a PMSM iron loss calculation model with broad applicability and high accuracy.

Two-timescale recurrent neural networks for distributed minimax optimization

In this paper, we present two-timescale neurodynamic optimization approaches to distributed minimax optimization. We propose four multilayer recurrent neural networks for solving four different types of generally nonlinear convex–concave minimax problems subject to linear equality and nonlinear inequality constraints. We derive sufficient conditions to guarantee the stability and optimality of the neural networks. We demonstrate the viability and efficiency of the proposed neural networks in two specific paradigms for Nash-equilibrium seeking in a zero-sum game and distributed constrained nonlinear optimization.

Fast Recognition of Snap-Fit for Industrial Robot Using a Recurrent Neural Network

Snap-fit recognition is an essential capability for industrial robots in manufacturing. The goal is to protect fragile parts by quickly detecting snap-fit signals in the assembly. In this letter, we propose a fast recognition method of snap-fit for industrial robots. A snap-fit dataset generation strategy of automatically acquiring labels is presented in the presence of data collection is complicated. A multilayer recurrent neural network (RNN) is designed for snap-fit recognition. An extensive evaluation based on two different datasets shows that the proposed method makes reliable and fast recognitions. Real-time experiments on industrial robot also demonstrate the effectiveness of the proposed method.

Short‐term load forecasting of multi‐scale recurrent neural networks based on residual structure

SummaryAccurate short‐term load forecasting plays an important role in reducing power generation costs, maintaining supply and demand balance, and stabling the power grids operation. In recent years, deep learning models based on recurrent neural networks (RNN) have been widely used in short‐term load forecasting. Nevertheless, RNN cannot extract multi‐scale features of load data, resulting in low forecasting accuracy. A model for short‐term power load forecasting of residual multiscale‐RNN (RM‐RNN) was proposed in this study. RM‐RNN uses the multilayer RNN network structure. Specifically, each layer sets the dilated convolution with different dilated coefficients to extract the multi‐scale features of the load data. Adjacent networks transfer feature information for feature fusion through the residual structures. The experiment used random sampling data training model, and compared RM‐RNN with multiple deep learning models. The experimental results demonstrated that the mean error of RM‐RNN prediction is the lowest, indicating that dilated convolution can effectively extract multi‐scale features of load data. This result verified the effectiveness of residual structure fusion features, and improved the accuracy of short‐term load forecasting.

A vector-to-sequence based multilayer recurrent network surrogate model for history matching of large-scale reservoir

History matching can estimate the parameter of spatially varying geological properties and provide reliable numerical models for reservoir development and management. However, in practice, high-dimension, multiple-solutions and computational cost are key issues that restrict the application of history matching methods. Recently, the combination of deep-learning-based surrogate model and sampling algorithm has been widely studied in history matching to overcome the limitations. Considering that real-world large-scale reservoirs often have hundreds of thousands or even millions of grid-based uncertain parameters, extracting spatial features using convolutional neural networks requires a lot of computational cost and storage requirements. Therefore, in this work, we mainly study how to use the recurrent neural network (RNN) to construct the surrogate model for history matching. Specifically, we propose a multilayer RNN surrogate model based on a vector-to-sequence modeling framework. The multilayer RNN surrogate model with gated recurrent unit (GRU), termed MLGRU, is developed to approximate the mapping from feature vector of geological realizations to the production data. The feature vector is the low-dimensional representation of geological parameter fields after using the re-parameterization method, while production data are the simulation results of historical period. In addition, we design a log-transformation-based windowed normalization (LTWN) method for the production data, which can enhance the learnability and features of production data. The MLGRU model is incorporated into a multimodal estimation of distribution algorithm (MEDA) to formulate a history matching workflow. The hyperparameters and performance of the proposed MLGRU model are analyzed by numerical experiments on a 2D reservoir model. Furthermore, numerical experiments performed on the Brugge benchmark model, a large-scale 3D reservoir model, demonstrated the performance of the proposed surrogate model and history matching method.

Parallel Gated Recurrent Unit Networks as an Encoder for Speech Recognition

Listen, Attend and Spell (LAS) network is one of the end-to-end approaches for speech recognition, which does not require an explicit language model. It consists of two parts; the encoder part which receives acoustic features as inputs, and the decoder network which produces one character at a time step, based on the encoder output and an attention mechanism. Multi-layer recurrent neural networks (RNN) are used in both decoder and encoder parts. Hence, the LAS architecture can be simplified as one RNN for the decoder, and another RNN for the encoder. Their shapes and layer sizes can be different. In this work, we examined the performance of using multi RNNs for the encoder part. Our baseline LAS network uses an RNN with a hidden size of 256. We used 2 and 4 RNNs with hidden sizes of 128 and 64 for each case. The main idea behind the proposed approach is to focus the RNNs to different patterns (phonemes in this case) in the data. At the output of the encoder, their outputs are concatenated and fed to the decoder. TIMIT database is used to compare the performance of the mentioned networks, using phoneme error rate as the performance metric. The experimental results showed that proposed approach can achieve a better performance than the baseline network. However, increasing the number of RNNs does not guarantee further improvements.

Data streams classification using deep learning under different speeds and drifts

Abstract Processing data streams arriving at high speed requires the development of models that can provide fast and accurate predictions. Although deep neural networks are the state-of-the-art for many machine learning tasks, their performance in real-time data streaming scenarios is a research area that has not yet been fully addressed. Nevertheless, much effort has been put into the adaption of complex deep learning (DL) models to streaming tasks by reducing the processing time. The design of the asynchronous dual-pipeline DL framework allows making predictions of incoming instances and updating the model simultaneously, using two separate layers. The aim of this work is to assess the performance of different types of DL architectures for data streaming classification using this framework. We evaluate models such as multi-layer perceptrons, recurrent, convolutional and temporal convolutional neural networks over several time series datasets that are simulated as streams at different speeds. In addition, we evaluate how the different architectures react to concept drifts typically found in evolving data streams. The obtained results indicate that convolutional architectures achieve a higher performance in terms of accuracy and efficiency, but are also the most sensitive to concept drifts.

Statistical Feature Construction for Forecasting Accuracy Increase and Its Applications in Neural Network Based Analysis

This paper presents a feature construction approach called Statistical Feature Construction (SFC) for time series prediction. Creation of new features is based on statistical characteristics of analyzed data series. First, the initial data are transformed into an array of short pseudo-stationary windows. For each window, a statistical model is created and characteristics of these models are later used as additional features for a single window or as time-dependent features for the entire time series. To demonstrate the effect of SFC, five plasma physics and six oceanographic time series were analyzed. For each window, unknown distribution parameters were estimated with the method of moving separation of finite normal mixtures. First four statistical moments of these mixtures for initial data and increments were used as additional data features. Multi-layer recurrent neural networks were trained to create short- and medium-term forecasts with a single window as input data; additional features were used to initialize the hidden state of recurrent layers. A hyperparameter grid-search was performed to compare fully-optimized neural networks for original and enriched data. A significant decrease in RMSE metric was observed with a median of 11.4%. There was no increase in RMSE metric in any of the analyzed time series. The experimental results have shown that SFC can be a valuable method for forecasting accuracy improvement.

Lyapunov stable learning laws for multilayer recurrent neural networks

This study aims to develop stable learning laws (in the sense of Lyapunov) for a general class of multilayer recurrent neural network (RNN) non-parametric identifier. The application of control Lyapunov functions in the discrete-time domain ensures the ultimate boundedness for the identification error enforcing it to reach a boundary set related to the power of parametric uncertainties and the modeling error. This work presents a general algorithm to design RNNs with n layers: one input layer, n-2 hidden layers, and one output layer. Numerical simulations support the theoretical results showing the advantages of increasing the number of layers in an RNN structure. A first example identifies the unknown states of the Van Der Pol oscillator. Then, a second example demonstrates the behavior of an RNN in a third-order system with high nonlinear dynamics describing an ozonation system of a single contaminant. For both cases, the application of the developed learning laws succeeds in estimating the uncertain dynamics. Moreover, the numerical simulations show how the identification error decreases as the number of layers increases.

Adaptive ensemble models for medium-term forecasting of water inflow when planning electricity generation under climate change

Medium-term forecasting of water inflow is of great importance for small hydroelectric power plants operating in remote power supply areas and having a small reservoir. Improving the forecasting accuracy is aimed at solving the problem of determining the water reserve for the future generation of electricity at hydroelectric power plants, taking into account the regulation in the medium term. Medium-term regulation is necessary to amplify the load in the peak and semi-peak portions of the load curve. The solution to such problems is aggravated by the lack of sufficiently reliable information on water inflow and prospective power consumption, which is of a stochastic nature. In addition, the mid-term planning of electricity generation should consider the seasonality of changes in water inflow, which directly affects the reserves and the possibility of regulation. The paper considers the problem of constructing a model for medium-term forecasting of water inflow for planning electricity generation, taking into account climatic changes in isolated power systems. Taking into account the regularly increasing effect of climate change, the current study proposes using an approach based on machine learning methods, which are distinguished by a high degree of autonomy and automation of learning, that is, the ability to self-adapt. The results showed that the error (RMSE) of the model based on the ensemble of regression decision trees due to constant self-adaptation decreased from 4.5 m3/s to 4.0 m3/s and turned out to be lower than the error of a more complex multilayer recurrent neural network (4.9 m3/s). The research results are intended to improve forecasting reliability in the planning, management, and operation of isolated operating power systems.

Prediction of InSAR deformation time-series using a long short-term memory neural network

ABSTRACT The prediction of land subsidence is a crucial step for early warning of urban infrastructure damage and timely remedy. However, the performance of most mathematical and empirical prediction models is often compromised by their large number of parameters, complex operational processes and sparsely measured values. Currently, the traditional neural network models are popular and effective, but they cannot accurately discover the characteristic changes of time series data. In this paper, a long short-term memory (LSTM) neural network was proposed to predict the land subsidence of time series Interferometric Synthetic Aperture Radar (InSAR). First, the Persistent Scatterer Interferometric Synthetic Aperture Radar (PS-InSAR) technique was utilized to monitor the time series land subsidence at Beijing Capital International Airport (BCIA) from 2005 to 2010 based on ENVISAT ASAR images with a descending orbit. The results were compared with the existing results to verify the reliability and then used to analyse the temporal and spatial characteristics of the time series land subsidence of the BCIA. Based on the time series InSAR deformation data, the LSTM neural network was used to establish the prediction model of time series InSAR, and the results were compared with those of the Multi-Layer Perceptron (MLP) and Recurrent Neural Network (RNN). The comparison results showed that the LSTM neural network was more accurate than the MLP and RNN on the point scale (the root mean square error was 4.60 mm and the mean absolute error was 3.18 mm), the correlation coefficients between the prediction results of the LSTM neural network and the real InSAR measurement results in 2007 and 2008 were 0.93 mm and 0.96 mm, respectively, indicating that LSTM neural network had better prediction performance. Eventually, based on the land subsidence data of time series InSAR from 2006 to 2010, the LSTM neural network was applied to predict the BCIA time series land subsidence in 2011. The results predicted that cumulative subsidence in September 2011 would reach a maximum of 350 mm. Therefore, the LSTM neural network is a potentially effective prediction method, which can replace numerical or empirical models in the absence of detailed hydrogeological data. Moreover, its prediction results can be used to assist decision-making, early warning and hazard relief.

Multi-Sensor Guided Hand Gesture Recognition for a Teleoperated Robot Using a Recurrent Neural Network

Touch-free guided hand gesture recognition for human-robot interactions plays an increasingly significant role in teleoperated surgical robot systems. Indeed, despite depth cameras provide more practical information for recognition accuracy enhancement, the instability and computational burden of depth data represent a tricky problem. In this letter, we propose a novel multi-sensor guided hand gesture recognition system for surgical robot teleoperation. A multi-sensor data fusion model is designed for performing interference in the presence of occlusions. A multilayer Recurrent Neural Network (RNN) consisting of a Long Short-Term Memory (LSTM) module and a dropout layer (LSTM-RNN) is proposed for multiple hand gestures classification. Detected hand gestures are used to perform a set of human-robot collaboration tasks on a surgical robot platform. Classification performance and prediction time is compared among the LSTM-RNN model and several traditional Machine Learning (ML) algorithms, such as k-Nearest Neighbor (k-NN) and Support Vector Machines (SVM). Results show that the proposed LSTM-RNN classifier is able to achieve a higher recognition rate and faster inference speed. In addition, the present adaptive data fusion system shows a strong anti-interference capability for hand gesture recognition in real-time.

Modeling of Psychomotor Reactions of a Person Based on Modification of the Tapping Test

The paper considers the method for analysis of a psychophysical state of a person on psychomotor indicators – finger tapping test. The app for mobile phone that generalizes the classic tapping test is developed for experiments. Developed tool allows collecting samples and analyzing them like individual experiments and like dataset as a whole. The data based on statistical methods and optimization of hyperparameters is investigated for anomalies, and an algorithm for reducing their number is developed. The machine learning model is used to predict different features of the dataset. These experiments demonstrate the data structure obtained using finger tapping test. As a result, we gained knowledge of how to conduct experiments for better generalization of the model in future. A method for removing anomalies is developed and it can be used in further research to increase an accuracy of the model. Developed model is a multilayer recurrent neural network that works well with the classification of time series. Error of model learning on a synthetic dataset is 1.5% and on a real data from similar distribution is 5%.

Malicious URL Detection Based on Improved Multilayer Recurrent Convolutional Neural Network Model

The traditional malicious uniform resource locator (URL) detection method excessively relies on the matching rules formulated by the network security personnel, which is hard to fully express the text information of the URL. Thus, an improved multilayer recurrent convolutional neural network model based on the YOLO algorithm is proposed to detect malicious URL in this paper. First, single characters are mapped to dense vectors using word embedding, and the dense vectors are participated in the training process of the whole model according to the structural characteristics of the URL in the method. Then, the CSPDarknet neural network model based on the improved YOLO algorithm is proposed to extract features of the URL. Finally, the extracted features are used to evaluate malicious URL by the bidirectional LSTM recurrent neural network algorithm. In order to verify the validity of the algorithm, a total of 200,000 URLs are collected, including 100,000 normal URLs labeled “good” and 100,000 malicious URLs labeled “bad”. The experimental results show that the method detects malicious URLs more quickly and effectively and has high accuracy, high recall rate, and high accuracy compared with Text-RCNN, BRNN, and other models.

Activity Recognition with Combination of Deeply Learned Visual Attention and Pose Estimation

While human activity recognition and pose estimation are closely related, these two issues are usually treated as separate tasks. In this thesis, two-dimension and three-dimension pose estimation is obtained for human activity recognition in a video sequence, and final activity is determined by combining it with an activity algorithm with visual attention. Two problems can be solved efficiently using a single architecture. It is also shown that end-to-end optimization leads to much higher accuracy than separated learning. The proposed architecture can be trained seamlessly with different categories of data. For visual attention, soft visual attention is used, and a multilayer recurrent neural network using long short term memory that can be used both temporally and spatially is used. The image, pose estimated skeleton, and RGB-based activity recognition data are all synthesized to determine the final activity to increase reliability. Visual attention evaluates the model in UCF-11 (Youtube Action), HMDB-51 and Hollywood2 data sets, and analyzes how to focus according to the scene and task the model is performing. Pose estimation and activity recognition are tested and analyzed on MPII, Human3.6M, Penn Action and NTU data sets. Test results are Penn Action 98.9%, NTU 87.9%, and NW-UCLA 88.6%.

Multistep Prediction of Bus Arrival Time with the Recurrent Neural Network

Accurate predictions of bus arrival times help passengers arrange their trips easily and flexibly and improve travel efficiency. Thus, it is important to manage and schedule the arrival times of buses for the efficient deployment of buses and to ease traffic congestion, which improves the service quality of the public transport system. However, due to many variables disturbing the scheduled transportation, accurate prediction is challenging. For accurate prediction of the arrival time of a bus, this research adopted a recurrent neural network (RNN). For the prediction, the variables affecting the bus arrival time were investigated from the data set containing the route, a driver, weather, and the schedule. Then, a stacked multilayer RNN model was created with the variables that were categorized into four groups. The RNN model with a separate multi-input and spatiotemporal sequence model was applied to the data of the arrival and leaving times of a bus from all of a Shandong Linyi bus route. The result of the model simulation revealed that the convolutional long short-term memory (ConvLSTM) model showed the highest accuracy among the tested models. The propagation of error and the number of prediction steps influenced the prediction accuracy.