Recurrent Neural Network Structure Research Articles

UFSP-Net: a neural network with spatio-temporal information fusion for urban fire situation prediction

Capturing the dynamics of urban fire situation is a basic but challenging task, which takes an indispensable role in the field of urban security and fire emergency decision. Traditional methods approach the urban fire prediction via stochastic process based on physics or statistics, which may be interpretable but less practical in real applications. Recently, some data-driven models, Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and Graph Convolutional Neural Network (GCN), seem to be fruitful in capturing spatio-temporal dynamics with massive high-dimension data. In this paper, we process some regional urban fire dataset of recent six years in the fire situation awareness images (FSAIs) and extract pixel-level latent representations with CNNs while GCNs are applied to process some spatial graph structure auxiliary information to obtain graph-level latent representations. And then Urban Fire Situation Prediction Neural Network (UFSP-Net) is formulated as a novel urban fire prediction model, integrating these two different kind of spatial latent representations and RNN structure. In comparison to other traditional algorithms, such as Conv-RNN, UFSP-Net demonstrates its superior prediction performance for multi type urban fire in spatio-temporal scale.

Process structure-based recurrent neural network modeling for model predictive control of nonlinear processes

In this work, physics-based recurrent neural network (RNN) modeling approaches are proposed for a general class of nonlinear dynamic process systems to improve prediction accuracy by incorporating a priori process knowledge. Specifically, a hybrid modeling method is first introduced to integrate first-principles models and RNN models. Subsequently, a partially-connected RNN modeling method that designs the RNN structure based on a priori structural process knowledge, and a weight-constrained RNN modeling method that employs weight constraints in the optimization problem of the RNN training process are developed. The proposed physics-based RNN models are utilized in model predictive controllers and applied to a chemical process network example to demonstrate their improved approximation performance compared to the fully-connected RNN model that is developed as a black box model.

Temporal Pyramid Recurrent Neural Network

Learning long-term and multi-scale dependencies in sequential data is a challenging task for recurrent neural networks (RNNs). In this paper, a novel RNN structure called temporal pyramid RNN (TP-RNN) is proposed to achieve these two goals. TP-RNN is a pyramid-like structure and generally has multiple layers. In each layer of the network, there are several sub-pyramids connected by a shortcut path to the output, which can efficiently aggregate historical information from hidden states and provide many gradient feedback short-paths. This avoids back-propagating through many hidden states as in usual RNNs. In particular, in the multi-layer structure of TP-RNN, the input sequence of the higher layer is a large-scale aggregated state sequence produced by the sub-pyramids in the previous layer, instead of the usual sequence of hidden states. In this way, TP-RNN can explicitly learn multi-scale dependencies with multi-scale input sequences of different layers, and shorten the input sequence and gradient feedback paths of each layer. This avoids the vanishing gradient problem in deep RNNs and allows the network to efficiently learn long-term dependencies. We evaluate TP-RNN on several sequence modeling tasks, including the masked addition problem, pixel-by-pixel image classification, signal recognition and speaker identification. Experimental results demonstrate that TP-RNN consistently outperforms existing RNNs for learning long-term and multi-scale dependencies in sequential data.

Operation optimization of a cryogenic NGL recovery unit using deep learning based surrogate modeling

In this work, the operation of a cryogenic expansion unit for the extraction of NGL is optimized through the implementation of data-driven techniques. The proposed approach is based on an optimization framework that integrates dynamic process simulations with two deep learning based surrogate models. The first model discloses the dynamics involved in the process using a long short-term memory (LSTM) layout with bidirectional recurrent neural network (RNN) structure and attention mechanism. The error maximization sampling strategy is adopted to improve the model accuracy. The second regression model is built to generate profit predictions of the process. Results from two case studies show the capabilities of the proposed optimization framework in terms of optimizing a cold residue reflux (CRR) NGL recovery unit.

Modeling of submerged membrane filtration processes using recurrent artificial neural networks

<span lang="EN-US">The modeling of membrane filtration processes is a challenging task because it involves many interactions from both biological and physical operational behavior. Membrane fouling behaviour in filtration processes is complex and hard to understand, and to derive a robust model is almost not possible. Therefore, it is the aim of this paper to study the potential of time series neural network based dynamic model for a submerged membrane filtration process. The developed model that represent the dynamic behavior of filtration process is later used in control design of the membrane filtration processes. In order to obtain the dynamic behaviour of permeate flux and transmembrane pressure (TMP), a random step was applied to the suction pump. A recurrent neural network (RNN) structure was employed to perform as the dynamic models of a filtration process, based on nonlinear auto-regressive with exogenous input (NARX) model structure. These models are compared with the linear auto-regressive with exogenous input (ARX) model. The performance of the models were evaluated in terms of %<em>R<sup>2</sup></em>, mean square error (MSE,) and a mean absolute deviation (MAD). For filtration control performance, a proportional integral derivative (PID) controller was implemented. The results showed that the RNN-NARX structure able to model the dynamic behavior of the filtration process under normal conditions in short range of the filtration process. The developed model can also be a reliable assistant for two different control strategies development in filtration processes.</span>

Dynamic Terminal Sliding-Mode Control for Single-Phase Active Power Filter Using New Feedback Recurrent Neural Network

In this article, an adaptive dynamic terminal sliding-mode controller using a double hidden layer recurrent neural network (DHL-RNN) structure for a single-phase active power filter (APF) is proposed to improve harmonic compensation performance. First, a method combining dynamic sliding mode and terminal sliding mode is proposed to solve the chattering phenomenon in traditional sliding-mode control. Then, since the nonlinear dynamics of APF is difficult to obtain accurately, the DHL-RNN is used to approximate the proposed dynamic terminal sliding-mode controller. Meanwhile, an integral robust switching term is added to eliminate the approximation error of the neural network. Simulation and experimental results proved that the proposed controller has better compensation performance and tracking effect compared with a simple terminal sliding-mode controller.

Recurrent neural network fractional-order sliding mode control of dynamic systems

In this study, a fractional-order sliding mode control (FSMC) scheme using a recurrent neural network (RNN) approximator is introduced to achieve better control performance for a class of dynamic systems. The proposed recurrent neural network fractional-order sliding mode control (RNNFSMC) scheme combines a fractional-order control method and a RNN structure. The primary objective of the proposed RNNFSMC scheme is to construct a fractional-order sliding mode controller. The RNN estimator is applied to approximate the unknown nonlinear part online. In the case study, the proposed RNNFSMC scheme is used for a dynamic model of an active power filter to realize the current harmonic compensation control. Simulation experiment proves that the proposed scheme is effective and robust and has a good tracking effect for dynamic system.

Short-Term Load Forecasting with Dilated Recurrent Attention Networks in Presence of Missing Data

Forecasting the dynamics of time-varying systems is essential to maintaining the sustainability of the systems.Recent studies have discovered that Recurrent Neural Networks(RNN) applied in the forecasting tasks outperform conventional models that include AutoRegressive Integrated Moving Average(ARIMA).However, due to the structural limitation of vanilla RNN which holds unit-length internal connections, learning the representation of time series with \textit{missing data} can be severely biased. The goal of this paper is to provide a robust RNN architecture against the bias from missing data. We propose Dilated Recurrent Attention Networks(DRAN).The proposed model has a stacked structure of multiple RNNs which layer of each having a different length of internal connections. This structure allows incorporating previous information at different time scales.DRAN updates its state by a weighted average of the layers.In order to focus more on the layer that carries reliable information against bias from missing data, it leverages attention mechanism which learns the distribution of attention weights among the layers.We report that our model outperforms conventional ones with respect to the forecast accuracy from two benchmark datasets, including a real-world electricity load dataset.

Learning deep hierarchical and temporal recurrent neural networks with residual learning

Learning both hierarchical and temporal dependencies can be crucial for recurrent neural networks (RNNs) to deeply understand sequences. To this end, a unified RNN framework is required that can ease the learning of both the deep hierarchical and temporal structures by allowing gradients to propagate back from both ends without being vanished. The residual learning (RL) has appeared as an effective and less-costly method to facilitate backward propagation of gradients. The significance of the RL is exclusively shown for learning deep hierarchical representations and temporal dependencies. Nevertheless, there is lack of efforts to unify these finding into a single framework for learning deep RNNs. In this study, we aim to prove that approximating identity mapping is crucial for optimizing both hierarchical and temporal structures. We propose a framework called hierarchical and temporal residual RNNs, to learn RNNs by approximating identity mappings across hierarchical and temporal structures. To validate the proposed method, we explore the efficacy of employing shortcut connections for training deep RNNs structures for sequence learning problems. Experiments are performed on Penn Treebank, Hutter Prize and IAM-OnDB datasets and results demonstrate the utility of the framework in terms of accuracy and computational complexity. We demonstrate that even for large datasets exploiting parameters for increasing network depth can gain computational benefits with reduced size of the RNN "state".

Conference Overview and Papers Program

To improve the workout efficiency and to provide the body movement suggestions to users in a “smart gym” environment, we propose to use a depth camera for capturing a user’s body parts and mount multiple inertial sensors on the body parts of a user to generate deadlift behavior models generated by a recurrent neural network structure. The contribution of this paper is trifold: 1) The multimodal sensing signals obtained from multiple devices are fused for generating the deadlift behavior classifiers, 2) the recurrent neural network structure can analyze the information from the synchronized skeletal and inertial sensing data, and 3) a Vaplab dataset is generated for evaluating the deadlift behaviors recognizing capability in the proposed method.

Variational neural decoder for abstractive text summarization

In the conventional sequence-to-sequence (seq2seq) model for abstractive summarization, the internal transformation structure of recurrent neural networks (RNNs) is completely determined. Therefore, the learned semantic information is far from enough to represent all semantic details and context dependencies, resulting in a redundant summary and poor consistency. In this paper, we propose a variational neural decoder text summarization model (VND). The model introduces a series of implicit variables by combining variational RNN and variational autoencoder, which is used to capture complex semantic representation at each step of decoding. It includes a standard RNN layer and a variational RNN layer [5]. These two network layers respectively generate a deterministic hidden state and a random hidden state. We use these two RNN layers to establish the dependence between implicit variables between adjacent time steps. In this way, the model structure can better capture the complex semantics and the strong dependence between the adjacent time steps when outputting the summary, thereby improving the performance of generating the summary. The experimental results show that, on the text summary LCSTS and English Gigaword dataset, our model has a significant improvement over the baseline model.

Double-Hidden-Layer Recurrent Neural Network Fractional-Order Sliding Mode Control of Shunt Active Power Filter

In this paper, a fractional-order sliding mode control scheme based on a double-hidden-layer recurrent neural network is proposed for a single-phase shunt active power filter. Considering the shortcomings of traditional neural networks that the approximation accuracy is not high and weight and center vector of neural networks are unchangeable, a new double-hidden-layer recurrent neural network structure which contains two hidden layers to make the network have more powerful fitting ability, is designed to approximate the unknown nonlinearities. An original output feedback neural network with two hidden layers is designed to estimate the uncertainties regardless of unknown system characteristics and external disturbances. A fractional-order term is added to the sliding mode controller to have more adjustable space and better optimization space. Experimental results verified the validity of the designed controller and proved that it can complete the current compensation well with acceptable current tracking error, demonstrating the outstanding compensation performance and strong robustness

Economic MPC of Nonlinear Processes via Recurrent Neural Networks Using Structural Process Knowledge

This work proposes three methods that incorporate a priori process knowledge into recurrent neural network (RNN) modeling of nonlinear processes to improve prediction accuracy and provide insights on the structure of neural network models. Specifically, we discuss a hybrid modeling method that integrates first-principles models and RNNs together, a partially-connected RNN modeling method that designs the RNN structure based on a priori structural process knowledge, and a weight-constrained RNN modeling method that introduces weight constraints in the optimization problem of RNN model training, respectively. The proposed RNN modeling methods are applied in the context of economic model predictive control of a chemical process example to demonstrate their improved approximation performance compared to a fully-connected RNN model that is developed as a black box model.

Microgrid Equivalent Modeling Based on Long Short-Term Memory Neural Network

Models of electrical equipment components are the basis of the transient stability studies of power system with multi-microgrids. Microgrid is a small local power system contains different electrical components which connected into distribution network through the Point of Common Couple(PCC). In order to simplify the grid-connect model of microgrid in power system stability study, a data-driven equivalent modeling method for microgrid based on Long Short-Term Memory(LSTM) recurrent neural network is proposed in this paper. LSTM recurrent neural network is the state-of-the-art models for a variety machine learning problems, and the dynamic behaviors of microgrid under grid-connect mode are presented by the LSTM recurrent neural network. The training data set should contain the dynamic behaviors of all components, the current and power of PCC are collected as the training data set during the faults for the parameter estimation of LSTM. Based on the measurement data set, a LSTM recurrent neural network structure with four inputs and two outputs is designed. During the training process, the real and imaginary parts of the current at present and previous time are taken as the input of the network, and errors of the output active and reactive power between equivalent model and detailed model of microgrid is taken as the evaluation index of the equivalent model method. An AC microgrid contains different distributed generations and load is built in DIgSILENT, and the accuracy of the proposed method is verified by comparing the equivalent model with the original detailed system.

A deep learning approach to conflating heterogeneous geospatial data for corn yield estimation: A case study of the US Corn Belt at the county level.

Understanding large-scale crop growth and its responses to climate change are critical for yield estimation and prediction, especially under the increased frequency of extreme climate and weather events. County-level corn phenology varies spatially and interannually across the Corn Belt in the United States, where precipitation and heat stress presents a temporal pattern among growth phases (GPs) and vary interannually. In this study, we developed a long short-term memory (LSTM) model that integrates heterogeneous crop phenology, meteorology, and remote sensing data to estimate county-level corn yields. By conflating heterogeneous phenology-based remote sensing and meteorological indices, the LSTM model accounted for 76% of yield variations across the Corn Belt, improved from 39% of yield variations explained by phenology-based meteorological indices alone. The LSTM model outperformed least absolute shrinkage and selection operator (LASSO) regression and random forest (RF) approaches for end-of-the-season yield estimation, as a result of its recurrent neural network structure that can incorporate cumulative and nonlinear relationships between corn yield and environmental factors. The results showed that the period from silking to dough was most critical for crop yield estimation. The LSTM model presented a robust yield estimation under extreme weather events in 2012, which reduced the root-mean-square error to 1.47Mg/ha from 1.93Mg/ha for LASSO and 2.43Mg/ha for RF. The LSTM model has the capability to learn general patterns from high-dimensional (spectral, spatial, and temporal) input features to achieve a robust county-level crop yield estimation. This deep learning approach holds great promise for better understanding the global condition of crop growth based on publicly available remote sensing and meteorological data.

Scalable Learning With a Structural Recurrent Neural Network for Short-Term Traffic Prediction

This paper presents a scalable deep learning approach for short-term traffic prediction based on historical traffic data in a vehicular road network. Capturing the spatio-temporal relationship of the big data often requires a significant amount of computational burden or an ad-hoc design aiming for a specific type of road network. To tackle the problem, we combine a road network graph with recurrent neural networks (RNNs) to construct a structural RNN (SRNN). The SRNN employs a spatio-temporal graph to infer the interaction between adjacent road segments as well as the temporal dynamics of the time series data. The model is scalable thanks to two key aspects. First, the proposed SRNN architecture is built by using the semantic similarity of the spatio-temporal dynamic interactions of all segments. Second, we design the architecture to deal with fixed-length tensors regardless of the graph topology. With the real traffic speed data measured in the city of Santander, we demonstrate the proposed SRNN outperforms the image-based approaches using the capsule network (CapsNet) by 14.1% and the convolutional neural network (CNN) by 5.87%, respectively, in terms of root mean squared error (RMSE). Moreover, we show that the proposed model is scalable. The SRNN model trained with data of a road network is able to predict traffic speed of different road networks, with the fixed number of parameters to train.

A Method for Identification of External Disturbing Influences on Electric Drives of a Large Radio Telescope

It is proposed to apply a neural network observer in the problems of identification of an external disturbance in the form of a wind load acting on the main mirror of a large radio telescope. The mathematical description of the wind moment is analyzed taking into account its fluctuating component, which has a nonlinear random character. A method of constructing the structure of recurrent neural networks based on models of a nonlinear autoregressive with exogenous inputs for identification of nonlinear static and dynamic control objects is described. In computer simulation, the RMS error for different training algorithms is compared to study their identification accuracy. Design, training, and testing of recurrent neural networks is carried out in the MATLAB/Simulink software environment.

Experimental Investigation of Recurrent Neural Network Fractional-Order Sliding Mode Control of Active Power Filter

In this brief, a fractional-order sliding mode control (FSMC) scheme using a recurrent neural network (RNN) approximator is introduced to achieve better performance for a shunt active power filter (APF). The proposed RNNFSMC scheme combines a fractional-order sliding mode control method with a recurrent neural network structure. The fractional-order sliding mode control has more adjustable degree of freedom to brings more superior control effect than integer order sliding mode control. The RNN estimator is employed to approximate the unknown nonlinear function of the APF. Experimental results are presented to show the effectiveness of the proposed strategy, demonstrating the outstanding compensation performance and strong robustness compared with standard neural sliding mode controller.

Online sales prediction via trend alignment-based multitask recurrent neural networks

While business trends are constantly evolving, the timely prediction of sales volume offers precious information for companies to achieve a healthy balance between supply and demand. In practice, sales prediction is formulated as a time series prediction problem which aims to predict the future sales volume for different products with the observation of various influential factors (e.g. brand, season, discount, etc.) and corresponding historical sales records. To perform accurate sales prediction under the offline setting, we gain insights from the encoder–decoder recurrent neural network (RNN) structure and have proposed a novel framework named TADA (Chen et al., in: ICDM, 2018) to carry out trend alignment with dual-attention, multitask RNNs for sales prediction. However, the sales data accumulates at a fast rate and is updated on a regular basis, rendering it difficult for the trained model to maintain the prediction accuracy with new data. In this light, we further extend the model into TADA$$^+$$, which is enhanced by an online learning module based on our innovative similarity-based reservoir. To construct the data reservoir for model retraining, different from most existing random sampling-based reservoir, our similarity-based reservoir selects data samples that are “hard” for the model to mine apparent dynamic patterns. The experimental results on two real-world datasets comprehensively show the superiority of TADA and TADA$$^+$$ in both online and offline sales prediction tasks against other state-of-the-art competitors.

FnnmOS-ELM: A Flexible Neural Network Mixed Online Sequential Elm

The learning speed of online sequential extreme learning machine (OS-ELM) algorithms is much higher than that of convolutional neural networks (CNNs) or recurrent neural network (RNNs) on regression and simple classification datasets. However, the general feature extraction of OS-ELM makes it difficult to conveniently and effectively perform classification on some large and complex datasets, e.g., CIFAR. In this paper, we propose a flexible OS-ELM-mixed neural network, termed as fnnmOS-ELM. In this mixed structure, the OS-ELM can replace a part of fully connected layers in CNNs or RNNs. Our framework not only exploits the strong feature representation of CNNs or RNNs, but also performs at a fast speed in terms of classification. Additionally, it avoids the problem of long training time and large parameter size of CNNs or RNNs to some extent. Further, we propose a method for optimizing network performance by splicing OS-ELM after CNN or RNN structures. Iris, IMDb, CIFAR-10, and CIFAR-100 datasets are employed to verify the performance of the fnnmOS-ELM. The relationship between hyper-parameters and the performance of the fnnmOS-ELM is explored, which sheds light on the optimization of network performance. Finally, the experimental results demonstrate that the fnnmOS-ELM has a stronger feature representation and higher classification performance than contemporary methods.