Dynamic Recurrent Neural Network Research Articles

A general dynamic sequential learning framework for vehicle trajectory reconstruction using automatic vehicle location or identification data

Vehicle trajectory data derived from automatic vehicle location (AVL) and automatic vehicle identification (AVI) systems provide critical support for intelligent transportation systems. However, the field-obtained vehicle trajectories are usually incomplete due to sensor malfunction or communication issues. To recover the incomplete data, the existing reconstruction methods have to impose strong assumptions on driver route choice behaviors (network level) and/or traffic dynamics (link level). With the tremendous data available, leveraging data-driven approaches to address the vehicle trajectory reconstruction problem with minimal assumptions is promising. This paper proposes a general dynamic sequential learning framework to reconstruct vehicle trajectory points for both AVL and AVI data. First, an Isolation Forest based ensemble learning model is developed to extract trajectory sequences attributed to different trips in an entire trip chain of a vehicle. Second, the dynamic recurrent neural network (dynamic RNN) is tailored to learn the underlying patterns from the complete AVL and AVI trajectories, respectively. Third, a sequential prediction scheme is customized to reconstruct AVL and AVI trajectories based on the trained networks. To validate the proposed method, two experiments are conducted. One is a simulation experiment with AVL data gathered from a well-calibrated simulation model. The other is a field experiment with AVI data collected from a real-world automatic license plate recognition (ALPR) system. The results show that the proposed method achieves superior performance for both AVL and AVI data compared with the traditional methods. The impacts of different sampling rates and traffic conditions on the model performance are also discussed.

Novel hybrid multi-head self-attention and multifractal algorithm for non-stationary time series prediction

Traditional time series prediction methods have shown their outstanding capabilities in time series prediction. However, due to essential differences in volatility characteristics among diverse types of non-stationary multivariate time series (NSMTS), it is difficult for traditional methods to maintain robust prediction performance. This study proposes a novel dynamic recurrent neural network to achieve stable and robust prediction performance. First, a multifractal gated recurrent unit (MF-GRU) based on the multifractal method is proposed to extract volatility characteristics. Meanwhile, to strengthen the parameters of the historical hidden layer state that has a more significant impact on the output, a self-attention mechanism is introduced into the MF-GRU, leading to a multifractal gated recurrent unit multi-head self-attention model. The efficiency of the proposed model was verified on public datasets. The experimental results show that the proposed model outperforms the traditional methods, such as long short-term memory (LSTM), the gated recurrent unit (GRU), and the minimal gated unit (MGU). etc.

Use of multilayer recursive model for non-linear dynamic system identification

In practice, the dynamics of the system are uncertain due to nonlinear and dynamic characteristics. It is difficult to establish accurate identification and prediction of the nonlinear plants that require dynamic modelling of the system. Extreme learning machine (ELM) as the recursive model due to its fast training and convergence speed is utilized in this work. However, its limitation is that it has only 1 hidden neuron which tends to make evolution speed low. Further, Multi-layer ELM (ML-ELM) model is applied on a nonlinear Auto-regressive complex benchmark system. The performance of ML-ELM is compared with dynamic recurrent functional link neural network (DRFLNN), functional link neural network (FLNN), nonlinear auto-regressive moving average (NARAX), multi-layer perception (MLP), radial basis function network (RBFN), Elman recurrent neural network (ERNN), and basic ELM models. From the comparison table, it can be seen that ML-ELM has better performance as compared with other models.

Robustness analysis and training of recurrent neural networks using dissipativity theory

Abstract Neural networks are widely applied in control applications, yet providing safety guarantees for neural networks is challenging due to their highly nonlinear nature. We provide a comprehensive introduction to the analysis of recurrent neural networks (RNNs) using robust control and dissipativity theory. Specifically, we consider H 2 {\mathcal{H}_{2}} -performance and the ℓ 2 {\ell _{2}} -gain to quantify the robustness of dynamic RNNs with respect to input perturbations. First, we analyze the robustness of RNNs using the proposed robustness certificates and then, we present linear matrix inequality constraints to be used in training of RNNs to enforce robustness. Finally, we illustrate in a numerical example that the proposed approach enhances the robustness of RNNs.

Prognostics for remaining useful life estimation in proton exchange membrane fuel cell by dynamic recurrent neural networks

This paper proposes a nonlinear dynamic recurrent neural network (DRNN) prognostics method for predicting the performance degradation trend and estimating the remaining useful life (RUL) of a proton exchange membrane fuel cell (PEMFC). The conducted DRNN prognostic methods are based on a nonlinear autoregressive neural network (NARNN) model and nonlinear autoregressive with exogenous inputs neural network (NARXNN) model, with the goal of taking actions to extend the life of the PEMFC. To effectively remove data noise or spikes and provide smooth reconstruction, a locally weighted regression (LWR) method is used for pre-processing. The DRNN model parameters for the time lags are processed by an autocorrelation function (ACF). For comparing the predictions within the different DRNN models, the data sets are divided into three different prognostic configurations for predicting and estimating the remaining life of the PEMFC. The results show that the NARXNN model has the best predictions for the degradation trend and accurate remaining useful life, i.e. better than the NARNN model. Finally, the prognostic capability of the proposed DRNN method is compared with those in the literature.

An innovative partition method for predicting shallow landslides by combining the slope stability analysis with a dynamic neural network model

The objective of this study is to explore the relevance between the rainfall pattern, slope stability of the soil layer, and the occurrence of shallow landslides. A seepage flow model for subsurface flow simulation with a slope stability analysis approach was established to examine the temporal and spatial variation of unstable grids. To define the threshold conditions to trigger shallow landslides for different regions in the watershed, the spatial distribution of the safety factor at the initial state was derived and adopted as the reference basis to classify all grids of the watershed into multiple zones. The first novelty of this study is to apply such a partition, depending on the watershed topography and soil characteristics, to train the landslide prediction model separately. The second novelty is to execute a dynamic recurrent neural network (RNN) model to determine the possible duration and the start time of shallow landslides for each zone by considering the rainfall condition as well as the cumulative area of unstable grids, which can be obtained by performing the seepage flow model. In this way, the physical significance of the RNN prediction model can be reinforced. The analysis results showed that the proposed methodology could effectively track the respective period of occurring shallow landslides for each zone, additionally, only some specific zones in the watershed were necessary to be investigated because they were prone to cause a large number of grids to become unstable during rainstorms.

Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks.

Lifetime learning, or the change (or acquisition) of behaviors during a lifetime, based on experience, is a hallmark of living organisms. Multiple mechanisms may be involved, but biological neural circuits have repeatedly demonstrated a vital role in the learning process. These neural circuits are recurrent, dynamic, and non-linear and models of neural circuits employed in neuroscience and neuroethology tend to involve, accordingly, continuous-time, non-linear, and recurrently interconnected components. Currently, the main approach for finding configurations of dynamical recurrent neural networks that demonstrate behaviors of interest is using stochastic search techniques, such as evolutionary algorithms. In an evolutionary algorithm, these dynamic recurrent neural networks are evolved to perform the behavior over multiple generations, through selection, inheritance, and mutation, across a population of solutions. Although, these systems can be evolved to exhibit lifetime learning behavior, there are no explicit rules built into these dynamic recurrent neural networks that facilitate learning during their lifetime (e.g., reward signals). In this work, we examine a biologically plausible lifetime learning mechanism for dynamical recurrent neural networks. We focus on a recently proposed reinforcement learning mechanism inspired by neuromodulatory reward signals and ongoing fluctuations in synaptic strengths. Specifically, we extend one of the best-studied and most-commonly used dynamic recurrent neural networks to incorporate the reinforcement learning mechanism. First, we demonstrate that this extended dynamical system (model and learning mechanism) can autonomously learn to perform a central pattern generation task. Second, we compare the robustness and efficiency of the reinforcement learning rules in relation to two baseline models, a random walk and a hill-climbing walk through parameter space. Third, we systematically study the effect of the different meta-parameters of the learning mechanism on the behavioral learning performance. Finally, we report on preliminary results exploring the generality and scalability of this learning mechanism for dynamical neural networks as well as directions for future work.

Architectural richness in deep reservoir computing

Reservoir computing (RC) is a popular class of recurrent neural networks (RNNs) with untrained dynamics. Recently, advancements on deep RC architectures have shown a great impact in time-series applications, showing a convenient trade-off between predictive performance and required training complexity. In this paper, we go more in depth into the analysis of untrained RNNs by studying the quality of recurrent dynamics developed by the layers of deep RC neural networks. We do so by assessing the richness of the neural representations in the different levels of the architecture, using measures originating from the fields of dynamical systems, numerical analysis and information theory. Our experiments, on both synthetic and real-world datasets, show that depth—as an architectural factor of RNNs design—has a natural effect on the quality of RNN dynamics (even without learning of the internal connections). The interplay between depth and the values of RC scaling hyper-parameters, especially the scaling of inter-layer connections, is crucial to design rich untrained recurrent neural systems.

A hybrid particle swarm optimization and recurrent dynamic neural network for multi-performance optimization of hard turning operation

Abstract In the present work, a new hybrid approach combining particle swarm optimization (PSO) algorithm with recurrent dynamic neural network (RDNN), which is described as PSO-RDNN algorithm, is proposed for multi-performance optimization of machining parameters in finish turning of hardened AISI D2. The suggested optimization problem is solved using the weighted sum technique. Process parameters including cutting speed and feed rate are optimized for minimizing operation cost, maximizing tool life, and producing parts with acceptable surface roughness. Based on experimental results, two neural network models were developed for predicting tool flank wear and surface roughness during the machining process. Based on trained neural networks and structured hybrid algorithm, optimum cutting parameters were obtained. The coefficient of determination for trained neural networks was calculated as R2 = 0.9893 and R2 = 0.9879 for predicted flank wear and surface roughness, respectively, which proves the efficiency of trained neural models in real industrial applications. Furthermore, the offered methodology returns a Pareto optimality graph, which represents optimized cutting variables for several various cutting conditions.

Implementation of a Commitment Machine for an Adaptive and Robust Expected Shortfall Estimation.

This study proposes a metaheuristic for the selection of models among different Expected Shortfall (ES) estimation methods. The proposed approach, denominated “Commitment Machine” (CM), has a strong focus on assets cross-correlation and allows to measure adaptively the ES, dynamically evaluating which is the most performing method through the minimization of a loss function. The CM algorithm compares four different ES estimation techniques which all take into account the interaction effects among assets: a Bayesian Vector autoregressive model, Stochastic Differential Equation (SDE) numerical schemes with Exponential Weighted Moving Average (EWMA), a Generalized AutoRegressive Conditional Heteroskedasticity (GARCH) volatility model and a hybrid method that integrates Dynamic Recurrent Neural Networks together with a Monte Carlo approach. The integration of traditional Monte Carlo approaches with Machine Learning technologies and the heterogeneity of dynamically selected methodologies lead to an improved estimation of the ES. The study describes the techniques adopted by the CM and the logic behind model selection; moreover, it provides a market application case of the proposed metaheuristic, by simulating an equally weighted multi-asset portfolio.

Short-term memory by transient oscillatory dynamics in recurrent neural networks

Despite the significance of short-term memory in cognitive function, the process of encoding and sustaining the input information in neural activity dynamics remains elusive. Herein, we unveiled the significance of transient neural dynamics to short-term memory. By training recurrent neural networks to short-term memory tasks and analyzing the dynamics, the characteristics of the short-term memory mechanism were obtained in which the input information was encoded in the amplitude of transient oscillations, rather than the stationary neural activities. This transient trajectory was attracted to a slow manifold, which permitted the discarding of irrelevant information. Additionally, we investigated the process by which the dynamics acquire robustness to noise. In this transient oscillation, the robustness to noise was obtained by a strong contraction of the neural states after perturbation onto the manifold. This mechanism works for several neural network models and tasks, which implies its relevance to neural information processing in general.

Towards developing a pocket therapist: an intelligent adaptive psychological support chatbot against mental health disorders in a pandemic situation

Nowadays with COVID-19 ongoing epidemic outbreak, containment for weeks was one of the most effective measures adopted to deal with the spread of the virus until a vaccine could be efficient. Over that period, increased anxiety, depression, suicide attempts and post-traumatic stress disorder are accumulated. Several studies referred to the need of using chatbots, which recognizes human emotions in such pandemic contexts. More recently, numerous research papers improved the ability of artificial intelligence methods to recognize human emotion. However, they are still limited. The aim of this paper is the development of a chatbot against the disturbing psychic consequences of the pandemic, taking human emotion recognition into account. The object is to help people; especially students; suffering from mental disorders, by progressively understanding the reasonsbehind them. This innovative chatbot was developed by using the natural language processing model of deep learning. An advanced model of deep learning has been elaborated the intention for people and that to help them to regulate their mood and to reduce distortion of negative thoughts, that why a collection of a new database was done. The sequence-to-sequence model encoder and decoder consist of Long short-term memory cells and it is defined with the bi-directional dynamic recurrent neural network packets.

EKF-DRNN autopilot for VLCC heading hybrid control

An online Extended Kalman Filter (EKF)-Dynamic Recurrent Neural Network (DRNN) autopilot implementation strategy for Very Large Crude Carrier (VLCC) heading hybrid control with uncertain dynamics is designed in this paper. The autopilot scheme is based on a DRNN control model, which learns VLCC dynamic characteristics, while the VLCC heading control is estimated by the EKF to minimize squared course error. The online EKF-DRNN autopilot provides optimal control on the basis of fuel-saving evaluation criteria using the heading deviation and rudder angle. Therefore, the autopilot output is guaranteed to converge to the desired VLCC trajectory asymptotically. The proposed strategy is evaluated by applying it to VLCC Yuan Kun Yang from COSCO Shipping, and works excellently under different loads, speed and weather conditions. The VLCC heading hybrid controller is also assessed by ‘Z’ manoeuvring and turning test, and the superiority of the online EKF-DRNN autopilot is demonstrated. The remote online monitoring of Yuan Kun Yang’s main navigation data shows that it improved fuel-saving properties despite worsening weather conditions causing increased yawing.

Use of Neural Network Based Prediction Algorithms for Powering Up Smart Portable Accessories

Emerging Trends in the use of smart portable accessories, particularly within the context of the Internet of Things (IoT), where smart sensor devices are employed for data gathering, require advancements in energy management mechanisms. This study aims to provide an intelligent energy management mechanism for wearable/portable devices through the use of predictions, monitoring, and analysis of the performance indicators for energy harvesting, majorly focusing on the hybrid PV-wind systems. To design a robust and precise model, prediction algorithms are compared and analysed for an efficient decision support system. Levenberg–Marquardt (LM), Bayesian Regularization (BR), and Scaled Conjugate Gradient (SCG) prediction algorithms are used to develop a Shallow Neural Network (SNN) for time series prediction. The proposed SNN model uses a closed-loop NARX recurrent dynamic neural network to predict the active power and energy of a hybrid system based on the experimental data of solar irradiation, wind speed, wind direction, humidity, precipitation, ambient temperature and atmospheric pressure collected from Jan 1st 2015 to Dec 26th 2015. The historical hourly metrological data set is established using calibrated sensors deployed at Middle East Technical University (METU), NCC. The accessory considered in this study is called as Smart Umbrella System (SUS), which uses a Raspberry Pi module to fetch the weather data from the current location and store it in the cloud to be processed using SNN classified prediction algorithms. The results obtained show that using the SNN model, it is possible to obtain predictions with 0.004 error rate in a computationally efficient way within 20 s. With the experiments, we are able to observe that for the period of observation, the energy harvested is 178 Wh/d, where the system estimates energy as 176.5 Wh/d, powering the portable accessories accurately.

RCNet: Incorporating Structural Information Into Deep RNN for Online MIMO-OFDM Symbol Detection With Limited Training

In this paper, we investigate online learning-based MIMO-OFDM symbol detection strategies focusing on a special recurrent neural network (RNN) - reservoir computing (RC). We first introduce the Time-Frequency RC to take advantage of the structural information inherent in OFDM signals. Using the time domain RC and the time-frequency RC as building blocks, we provide two extensions of the shallow RC to RCNet: 1) Stacking multiple time domain RCs; 2) Stacking multiple time-frequency RCs into a deep structure. The combination of RNN dynamics, the time-frequency structure of MIMO-OFDM signals, and the deep network enables RCNet to handle the interference and nonlinear distortion of MIMO-OFDM signals to outperform existing methods. Unlike most existing NN-based detection strategies, RCNet is also shown to provide a good generalization performance even with a limited online training set (i.e, similar amount of reference signals/training as standard model-based approaches). Numerical experiments demonstrate that the introduced RCNet can offer a faster learning convergence and as much as 20% gain in bit error rate over a shallow RC structure by compensating for the nonlinear distortion of the MIMO-OFDM signal, such as due to power amplifier compression in the transmitter or due to finite quantization resolution in the receiver.

Control and Diagnostics of TV3-117 Aircraft Engine Technical State in Flight Modes Using the Matrix Method for Calculating Dynamic Recurrent Neural Networks

Control and Diagnostics of TV3-117 Aircraft Engine Technical State in Flight Modes Using the Matrix Method for Calculating Dynamic Recurrent Neural Networks

A novel dynamic recurrent functional link neural network-based identification of nonlinear systems using Lyapunov stability analysis

In this paper, a novel dynamic recurrent functional link neural network (DRFLNN) is proposed for the identification of unknown dynamics of the nonlinear systems. The proposed structure contains a self-feedback loop(s) as well as the adjustable weighted feed-through of the input signals to the output neuron(s). A learning algorithm is developed using the combination of Lyapunov stability and dynamic back-propagation method and is applied to derive the stable parameter adjustment equations. The performance evaluation of the proposed DRFLNN model is done by comparing it with the multi-layer perceptron (consisting of a single hidden layer), radial basis function network, Elman recurrent neural network (ERNN), nonlinear auto-regressive moving average, and the conventional functional link neural network. Three benchmark systems have been used on which all these models are applied. From the results, it is found that ERNN provided better prediction accuracy as compared to the remaining models and the second-best accuracy is obtained from the proposed model. Further, the ERNN model is more complex and offers more parameters to be tuned as compared to the DRFLNN model. Thus, the training of the ERNN model is quite difficult as compared to the DRFLNN.

Character level and word level embedding with bidirectional LSTM – Dynamic recurrent neural network for biomedical named entity recognition from literature

Named Entity Recognition is the process of identifying different entities in a given context. Biomedical Named Entity Recognition (BNER) is the task of extracting chemical names from biomedical texts to support biomedical and translational research. The aim of the system is to extract useful chemical names from biomedical literature text without a lot of handcrafted engineering features. This approach introduces a novel neural network architecture with the composition of bidirectional long short-term memory (BLSTM), dynamic recurrent neural network (RNN) and conditional random field (CRF) that uses character level and word level embedding as the only features to identify the chemical entities. Using this approach we have achieved the F1 score of 89.98 on BioCreAtIvE II GM corpus and 90.84 on NCBI corpus by outperforming the existing systems. Our system is based on the deep neural architecture that uses both character and word level embedding which captures the morphological and orthographic information eliminating the need for handcrafted engineering features. The proposed system outperforms the existing systems without a lot of handcrafted engineering features. The embedding concept along with the bidirectional LSTM network proved to be an effective method to identify most of the chemical entities.

An Improved Elman Network for Stock Price Prediction Service

The rapid development of edge computing drives the rapid development of stock market prediction service in terminal equipment. However, the traditional prediction service algorithm is not applicable in terms of stability and efficiency. In view of this challenge, an improved Elman neural network is proposed in this paper. Elman neural network is a typical dynamic recurrent neural network that can be used to provide the stock price prediction service. First, the prediction model parameters and build process are analysed in detail. Then, the historical data of the closing price of Shanghai composite index and the opening price of Shenzhen composite index are collected for training and testing, so as to predict the prices of the next trading day. Finally, the experiment results validate that it is effective to predict the short-term future stock price by using the improved Elman neural network model.

A non-invasive continuous cuffless blood pressure estimation using dynamic Recurrent Neural Networks

Cardiovascular diseases (CVD) have become the most important health problem of our time. High blood pressure, which is cardiovascular disease, is a risk factor for death, stroke, and heart attack. Blood pressure measurement is commonly used to limit blood flow in the arm or wrist, with the cuff. Since blood pressure cannot be measured continuously in this method, the dynamics underlying blood pressure cannot be determined and are inefficient in capturing symptoms. This paper aims to perform blood pressure estimation using Photoplethysmography (PPG) and Electrocardiography (ECG) signals that do not obstruct the vascular access. These signals were filtered and segmented synchronously from the R interval of the ECG signal, and chaotic, time, and frequency domain features were subtracted, and estimation methods were applied. Different methods of machine learning in blood pressure estimation are compared. Dynamic learning methods such as Recurrent Neural Network (RNN), Nonlinear Autoregressive Network with Exogenous Inputs Neural Networks NARX-NN and Long-Short Term Memory Neural Network (LSTM-NN) used. Estimation results have been evaluated with performance criteria. Systolic Blood Pressure (SBP) error mean ± standard deviation = 0.0224 ± (2.211), Diastolic Blood Pressure (DBP) error mean ± standard deviation = 0.0417 ± (1.2193) values have been detected in NARX artificial neural network. The blood pressure estimation results are evaluated by the British Hypertension Society (BHS) and American National Standard for Medical Instrumentation ANSI/AAMI SP10: 2002. Finding the most accurate and easy method in blood pressure measurement will contribute to minimizing the errors.