Variant Of Recurrent Neural Network Research Articles

Cyclone trajectory and intensity prediction with uncertainty quantification using variational recurrent neural networks

Cyclone trajectory and intensity prediction with uncertainty quantification using variational recurrent neural networks

State of charge estimation of Li-ion batteries based on deep learning methods and particle-swarm-optimized Kalman filter

The estimation of SOC is a key issue for the high-efficient and reliable operation of Li-ion batteries, thus has been increasingly concerned in current years with the development of electric vehicles. During dynamic test cycles, the accurate estimation of SOC is more difficult than steady operation conditions due to the fierce oscillations of the input signals. In this paper, a hybrid method of deep learning method and Kalman filter was proposed for the estimation of SOC. First, convolutional neural network or temporal convolutional network was combined with different variants of recurrent neural network, including long short term memory, gated recurrent unit, peeple hole long short term memory and bidirectional long short term memory, to achieve the estimation of SOC by capturing the spatial and temporal characteristics of input signals. Afterwards, the deep learning method was integrated with Kalman filter to eliminate the effects of transient signal oscillations and further improve the accuracy for the estimation of SOC. The results indicated that estimation accuracy and estimation time could be improved by less than 20 % by varying deep learning methods while after integrating deep learning method with Kalman filter, more than 45 % improvements in test accuracy could be achieved without obvious sacrifices in estimation time.

NT-Com: A combined machine learning model for picking up first arrival

We present a combined model named NT-Com that could predict the noise of observation site in China and effectively pick up first arrivals with very high accuracy. Based on a nonlinear autoregressive exogenous model (NARX) known as a variant of recurrent neural network and classification-regression trees (CART), NT-Com picks first arrivals by detecting abrupt points in time series. Several experiments were performed to check the fitting degree and prediction accuracy of NARX on noise signals, and experiments on CART were made to measure the accuracy of first break picking. The experimental results of NT-Com are close to those of experts and better than the traditional automatic detection algorithms. Moreover, the accuracy of NT-Com can reach 96%. The time error of the P-wave first arrival pickup can be shortened to 0.5 s. Because various first arrival patterns are learned during the training process, they can be effectively identified, which leads to a much lower probability of false triggers than traditional automatic detection algorithms. Trained and tested by all seismic data in China Earthquake networks from 2009–2019, the research work finds out a solution for picking up first arrivals by combining a simple structure of the neural networks and trees, which can be applied in China with its strong generalization ability and robustness.

Emotion classification of Indonesian Tweets using Bidirectional LSTM

Emotion classification can be a powerful tool to derive narratives from social media data. Traditional machine learning models that perform emotion classification on Indonesian Twitter data exist but rely on closed-source features. Recurrent neural networks can meet or exceed the performance of state-of-the-art traditional machine learning techniques using exclusively open-source data and models. Specifically, these results show that recurrent neural network variants can produce more than an 8% gain in accuracy in comparison with logistic regression and SVM techniques and a 15% gain over random forest when using FastText embeddings. This research found a statistical significance in the performance of a single-layer bidirectional long short-term memory model over a two-layer stacked bidirectional long short-term memory model. This research also found that a single-layer bidirectional long short-term memory recurrent neural network met the performance of a state-of-the-art logistic regression model with supplemental closed-source features from a study by Saputri et al. [8] when classifying the emotion of Indonesian tweets.

Markovian RNN: An Adaptive Time Series Prediction Network With HMM-Based Switching for Nonstationary Environments.

We investigate nonlinear regression for nonstationary sequential data. In most real-life applications such as business domains including finance, retail, energy, and economy, time series data exhibit nonstationarity due to the temporally varying dynamics of the underlying system. We introduce a novel recurrent neural network (RNN) architecture, which adaptively switches between internal regimes in a Markovian way to model the nonstationary nature of the given data. Our model, Markovian RNN employs a hidden Markov model (HMM) for regime transitions, where each regime controls hidden state transitions of the recurrent cell independently. We jointly optimize the whole network in an end-to-end fashion. We demonstrate the significant performance gains compared to conventional methods such as Markov Switching ARIMA, RNN variants and recent statistical and deep learning-based methods through an extensive set of experiments with synthetic and real-life datasets. We also interpret the inferred parameters and regime belief values to analyze the underlying dynamics of the given sequences.

SwinVRNN: A Data‐Driven Ensemble Forecasting Model via Learned Distribution Perturbation

AbstractThe data‐driven approaches for medium‐range weather forecasting are recently shown to be extraordinarily promising for ensemble forecasting due to their fast inference speed compared to the traditional numerical weather prediction models. However, their forecast accuracy can hardly match the state‐of‐the‐art operational ECMWF Integrated Forecasting System (IFS) model. Previous data‐driven approaches perform ensemble forecasting using some simple perturbation methods, like the initial condition perturbation and the Monte Carlo dropout. However, their ensemble performance is often limited arguably by the sub‐optimal ways of applying perturbation. We propose a Swin Transformer‐based Variational Recurrent Neural Network (SwinVRNN), which is a stochastic weather forecasting model combining a SwinRNN predictor with a perturbation module. SwinRNN is designed as a Swin Transformer‐based recurrent neural network, which predicts the future states deterministically. Furthermore, to model the stochasticity in the prediction, we design a perturbation module following the Variational Auto‐Encoder paradigm to learn the multivariate Gaussian distributions of a time‐variant stochastic latent variable from the data. Ensemble forecasting can be easily performed by perturbing the model features leveraging the noise sampled from the learned distribution. We also compare four categories of perturbation methods for ensemble forecasting, that is, fixed distribution perturbation, learned distribution perturbation, MC dropout, and multi model ensemble. Comparisons on the WeatherBench data set show that the learned distribution perturbation method using our SwinVRNN model achieves remarkably improved forecasting accuracy and reasonable ensemble spread due to the joint optimization of the two targets. More notably, SwinVRNN surpasses operational IFS on the surface variables of the 2‐m temperature and the 6‐hourly total precipitation at all lead times up to 5 days (Code is available at https://github.com/tpys/wwprediction).

Robust recurrent neural networks for time series forecasting

Recurrent neural networks (RNNs) are widely utilized in time series forecasting tasks. In practical applications, there are noises in real-life time series data. A model’s generalization capacity will be diminished by model uncertainty with regard to input noises. However, the robustness of RNNs with respect to input noises has not been well studied yet. The localized stochastic sensitivity (LSS), which measures output disturbances with respect to input perturbations of learning models, has been successfully applied to improve the robustness of different neural networks on tabular and image data. But its effectiveness on time series data has not been explored. Therefore, we extend the idea of LSS and apply it to the robust RNNs training for time series forecasting problems. With the minimization of LSS, output sensitivities of RNNs with respect to small perturbations are reduced. So, the proposed robust RNNs will not be affected by slight input noises. We have used the LSTM as an example for theoretical analysis and analyzed the effectiveness of LSS on several RNN variants including the vanilla RNN, the gated recurrent unit (GRU), the long-short-term memory (LSTM), and the bi-directional long short-term memory (Bi-LSTM) in empirical studies. Experimental results confirm the efficiency of applying LSS to enhance the robustness of RNNs for time series data. For example, the robust RNNs outperform their counterpart by 53.26% and 49.45% on average in terms of Root Mean Squared Error and R-Square on five datasets.

Dynamical Hyperspectral Unmixing with Variational Recurrent Neural Networks.

Multitemporal hyperspectral unmixing (MTHU) is a fundamental tool in the analysis of hyperspectral image sequences. It reveals the dynamical evolution of the materials (endmembers) and of their proportions (abundances) in a given scene. However, adequately accounting for the spatial and temporal variability of the endmembers in MTHU is challenging, and has not been fully addressed so far in unsupervised frameworks. In this work, we propose an unsupervised MTHU algorithm based on variational recurrent neural networks. First, a stochastic model is proposed to represent both the dynamical evolution of the endmembers and their abundances, as well as the mixing process. Moreover, a new model based on a low-dimensional parametrization is used to represent spatial and temporal endmember variability, significantly reducing the amount of variables to be estimated. We propose to formulate MTHU as a Bayesian inference problem. However, the solution to this problem does not have an analytical solution due to the nonlinearity and non-Gaussianity of the model. Thus, we propose a solution based on deep variational inference, in which the posterior distribution of the estimated abundances and endmembers is represented by using a combination of recurrent neural networks and a physically motivated model. The parameters of the model are learned using stochastic backpropagation. Experimental results show that the proposed method outperforms state of the art MTHU algorithms.

Using Recurrent Neural Network Structure and Multi-Head Attention with Convolution for Fraudulent Phone Text Recognition

Fraud cases have been a risk in society and people’s property security has been greatly threatened. In recent studies, many promising algorithms have been developed for social media offensive text recognition as well as sentiment analysis. These algorithms are also suitable for fraudulent phone text recognition. Compared to these tasks, the semantics of fraudulent words are more complex and more difficult to distinguish. Recurrent Neural Networks (RNN), the variants of RNN, Convolutional Neural Networks (CNN), and hybrid neural networks to extract text features are used by most text classification research. However, a single network or a simple network combination cannot obtain rich characteristic knowledge of fraudulent phone texts relatively. Therefore, a new model is proposed in this paper. In the fraudulent phone text, the knowledge that can be learned by the model includes the sequence structure of sentences, the correlation between words, the correlation of contextual semantics, the feature of keywords in sentences, etc. The new model combines a bidirectional Long-Short Term Memory Neural Network (BiLSTM) or a bidirectional Gate Recurrent United (BiGRU) and a Multi-Head attention mechanism module with convolution. A normalization layer is added after the output of the final hidden layer. BiLSTM or BiGRU is used to build the encoding and decoding layer. Multi-head attention mechanism module with convolution (MHAC) enhances the ability of the model to learn global interaction information and multi-granularity local interaction information in fraudulent sentences. A fraudulent phone text dataset is produced by us in this paper. The THUCNews data sets and fraudulent phone text data sets are used in experiments. Experiment results show that compared with the baseline model, the proposed model (LMHACL) has the best experiment results in terms of Accuracy, Precision, Recall, and F1 score on the two data sets. And the performance indexes on fraudulent phone text data sets are all above 0.94.

A Hybrid Deep Fused Learning Approach to Segregate Infectious Diseases

Humankind is facing another deadliest pandemic of all times in history, caused by COVID-19. Apart from this challenging pandemic, World Health Organization (WHO) considers tuberculosis (TB) as a preeminent infectious disease due to its high infection rate. Generally, both TB and COVID-19 severely affect the lungs, thus hardening the job of medical practitioners who can often misidentify these diseases in the current situation. Therefore, the time of need calls for an immediate and meticulous automatic diagnostic tool that can accurately discriminate both diseases. As one of the preliminary smart health systems that examine three clinical states (COVID-19, TB, and normal cases), this study proposes an amalgam of image filtering, data-augmentation technique, transfer learning-based approach, and advanced deep-learning classifiers to effectively segregate these diseases. It first employed a generative adversarial network (GAN) and Crimmins speckle removal filter on X-ray images to overcome the issue of limited data and noise. Each pre-processed image is then converted into red, green, and blue (RGB) and Commission Internationale de l’Elcairage (CIE) color spaces from which deep fused features are formed by extracting relevant features using DenseNet121 and ResNet50. Each feature extractor extracts 1000 most useful features which are then fused and finally fed to two variants of recurrent neural network (RNN) classifiers for precise discrimination of three-clinical states. Comparative analysis showed that the proposed Bi-directional long-short-term-memory (Bi-LSTM) model dominated the long-short-term-memory (LSTM) network by attaining an overall accuracy of 98.22% for the three-class classification task, whereas LSTM hardly achieved 94.22% accuracy on the test dataset.

Job Recommendation Based on Recurrent Neural Network Approach

Employers couldn't find relevant candidates without great effort, and job seekers must spend a long time researching the right jobs. To fix this issue, recommender systems are an effective way to make talent and job research more effective and easier to complete. It is possible to analyze job offers and candidate profiles and make pertinent recommendations using techniques like the Recurrent Neural Network (RNN) approach. To accomplish this, we propose a model based on the classification of job profiles using a variant of RNN called LSTM ("Long short-term memory"). It is a long-term memory model based on a deep learning algorithm. To maintain the semantic meaning, first, the input of the LSTM model is transformed into a set of vectors using the doc2Vec method, which transforms a document into a vector of numbers.

Deep Dynamic Scene Deblurring From Optical Flow

Deblurring can not only provide visually more pleasant pictures and make photography more convenient, but also can improve the performance of objection detection as well as tracking. However, removing dynamic scene blur from images is a non-trivial task as it is difficult to model the non-uniform blur mathematically. Several methods first use single or multiple images to estimate optical flow (which is treated as an approximation of blur kernels) and then adopt non-blind deblurring algorithms to reconstruct the sharp images. However, these methods cannot be trained in an end-to-end manner and are usually computationally expensive. In this paper, we explore optical flow to remove dynamic scene blur by using the multi-scale spatially variant recurrent neural network (RNN). We utilize FlowNets to estimate optical flow from two consecutive images in different scales. The estimated optical flow provides the RNN weights in different scales so that the weights can better help RNNs to remove blur in the feature spaces. Finally, we develop a convolutional neural network (CNN) to restore the sharp images from the deblurred features. Both quantitatively and qualitatively evaluations on the benchmark datasets demonstrate that the proposed method performs favorably against state-of-the-art algorithms in terms of accuracy, speed, and model size.

DAFA-BiLSTM: Deep Autoregression Feature Augmented Bidirectional LSTM network for time series prediction

Time series forecasting models that use the past information of exogenous or endogenous sequences to forecast future series play an important role in the real world because most real-world time series datasets are rich in time-dependent information. Most conventional prediction models for time series datasets are time-consuming and fraught with complex limitations because they usually fail to adequately exploit the latent spatial dependence between pairs of variables. As a successful variant of recurrent neural networks, the long short-term memory network (LSTM) has been demonstrated to have stronger nonlinear dynamics to store sequential data than traditional machine learning models. Nevertheless, the common shallow LSTM architecture has limited capacity to fully extract the transient characteristics of long interval sequential datasets. In this study, a novel deep autoregression feature augmented bidirectional LSTM network (DAFA-BiLSTM) is proposed as a new deep BiLSTM architecture for time series prediction. Initially, the input vectors are fed into a vector autoregression (VA) transformation module to represent the time-delayed linear and nonlinear properties of the input signals in an unsupervised way. Then, the learned nonlinear combination vectors of VA are progressively fed into different layers of BiLSTM and the output of the previous BiLSTM module is also concatenated with the time-delayed linear vectors of the VA as an augmented feature to form new additional input signals for the next adjacent BiLSTM layer. Extensive real-world time series applications are addressed to demonstrate the superiority and robustness of the proposed DAFA-BiLSTM. Comparative experimental results and statistical analysis show that the proposed DAFA-BiLSTM has good adaptive performance as well as robustness even in noisy environment.

Latent Representation in Human–Robot Interaction With Explicit Consideration of Periodic Dynamics

This paper presents a new data-driven framework for analyzing periodic physical human-robot interaction (pHRI) in latent state space. To elaborate human understanding and/or robot control during pHRI, the model representing pHRI is critical. Recent developments of deep learning technologies would enable us to learn such a model from a dataset collected from the actual pHRI. Our framework is developed based on variational recurrent neural network (VRNN), which can inherently handle time-series data like one pHRI generates. This paper modifies VRNN in order to include the latent dynamics from robot to human explicitly. In addition, to analyze periodic motions like walking, we integrate a new recurrent network based on reservoir computing (RC), which has random and fixed connections between numerous neurons, with VRNN. By augmenting RC into complex domain, periodic behavior can be represented as the phase rotation in complex domain without decaying the amplitude. For verification of the proposed framework, a rope-rotation/swinging experiment was analyzed. The proposed framework, trained on the dataset collected from the experiment, achieved the latent state space where the differences in periodic motions can be distinguished. Such a well-distinguished space yielded the best prediction accuracy of the human observations and the robot actions. The attached video can be seen in youtube: https://youtu.be/umn0MVcIpsY

Common belief multi-agent reinforcement learning based on variational recurrent models

The tacit cooperation among human teams benefits from the fact that consensus can be reached on a task through common belief. Similar to human social groups, agents in distributed learning systems can also rely on common belief to achieve cooperation under the condition of limited communication. In this paper, we show the role of common belief among agents in completing cooperative tasks, by proposing the Common Belief Multi-Agent (CBMA) reinforcement learning method. CBMA is a novel value-based method that infers the belief between agents with a variational model and models the environment with a variational recurrent neural network. We validate CBMA on two grid-world games as well as the StarCraft II micromanagement benchmark. Experimental results show that the learned common belief by CBMA can improve performance in both discrete and continuous state settings.

TEXT CLASSIFICATION USING DEEP LEARNING

We introduce a novel approach for automatically classifying the sentiment of Twitter messages using NLP and deep learning. These messages are classified as either positive or negative with respect to a query term using sentiment analysis. This is useful for consumers who want to research the sentiment of products before purchase, or companies that want to monitor the public sentiment of their brands. There is no previous research on classifying sentiment of messages on microblogging services like Twitter. We present the results of deep learning algorithm for classifying the sentiment of Twitter messages using distant supervision. Our training data consists of Twitter messages. This type of training data is abundantly available and can be obtained through automated means. We show that deep learning algorithm LSTM which is a variant of recurrent neural network (RNN) have accuracy approximately 80% when trained with this data. This paper also describes the preprocessing steps needed in order to achieve high accuracy. The main contribution of this paper is the idea of using tweets for distant supervised learning.

DynaLAP: Human Activity Recognition in Fixed Protocols via Semi-Supervised Variational Recurrent Neural Networks With Dynamic Priors

Learning the route and order of tasks can be critical to human activity recognition (HAR) for fixed protocols of movement. In this article, we propose a novel framework, DynaLAP, a semi-supervised variational recurrent neural network (VRNN) with a dynamic prior distribution, to perform activity recognition in fixed protocols. DynaLAP takes single tri-axial accelerometry data as input and causally classifies the activity of 10–30-s windows at a time. DynaLAP learns not only a window-specific short-term state, but also a long-term dynamic state iteratively updated throughout the protocol’s measurements. Additionally, instead of using a stationary prior distribution of activity classes, DynaLAP learns a dynamic prior that updates for each window. DynaLAP thereby learns protocol-specific dynamics when trained on data from subjects abiding by a fixed protocol. Two datasets from previously published literature were used to evaluate DynaLAP: the fully labeled MotionSense dataset of 24 subjects and a weakly labeled dataset of 17 subjects collected at the Georgia Institute of Technology. For each dataset, we varied the number of training labels used from a single subject’s data to the entire dataset. DynaLAP outperformed previous supervised and semi-supervised HAR approaches by 6–42 percentage points, with F1 scores that remained above 80%. These results suggest that DynaLAP can achieve state-of-the-art HAR performance in fixed protocols by learning protocol-specific dynamics, especially in weakly and scarcely labeled settings. DynaLAP could ultimately reduce the necessity for labor-intensive annotation efforts in HAR applications involving routine activities (e.g., military training).

RNN‐EdgeQL: An auto‐scaling and placement approach for SFC

SummaryThis paper proposes a prediction‐based scaling and placement of service function chains (SFCs) to improve service level agreement (SLA) and reduce operation cost. We used a variant of recurrent neural network (RNN) called gated recurrent unit (GRU) for resource demand prediction. Then, considering these predictions, we built an intuitive scale in/out algorithm. We also developed an algorithm that applies Q‐Learning on Edge computing environment (EdgeQL) to place these scaled‐out VNFs in appropriate locations. The integrated algorithm that combines prediction, scaling, and placement are called RNN‐EdgeQL. RNN‐EdgeQL (v2) is further improved to achieve application agnostic group level elasticity in the chain, independent of applications installed on the VNFs. We tested our algorithm on two realistic temporal dynamic load models including Internet traffic (Abilene) and an application specific traffic (Wiki) on an OpenStack testbed. The contribution of this article is threefold. First, prediction model prepares the target SFC for the upcoming load. Second, an application agnostic characteristics of the algorithm achieves the group‐level elasticity in SFC. Finally, the EdgeQL placement model minimizes the end‐to‐end path of an SFC in multi‐access edge computing (MEC) environment. As a result, RNN‐EdgeQL (v2) gives the lowest overall latency, lowest SLA violations, and lowest VNFs requirement, compared to RNN‐EdgeQL (v1) and Threshold‐Openstack default placement.

GBC: An Energy-Efficient LSTM Accelerator With Gating Units Level Balanced Compression Strategy

Recurrent Neural Networks (RNNs) have emerged as one of the most popular neural networks for processing time-series problems, widely used in machine translation, automatic speech recognition, and other natural language processing applications. However, conventional RNNs suffered from vanishing and exploding gradients, resulting in poor network performance in applications with long-term input information. As a variant of RNN, Long Short-Term Memory (LSTM) had been proposed to tackle this issue. Nevertheless, at the same time, LSTM introduces gating units and many additional parameters, which makes it challenging to be implemented directly on resource-limited platforms, such as Field Programmable Gate Arrays (FPGAs). This work first investigated the overall maximum achievable compression rates of different gating units and their correlations. Then, Gating Units Level Balanced Compression (GBC) strategy is proposed. After Top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> pruning, the proposed GBC strategy can attain a compression rate of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$36.6\times $ </tex-math></inline-formula> for LSTM. Further, the theoretical analysis indicates that for the existing gating units level LSTM compression variants, the GBC strategy still has further potential for compression. A complementary compression of the GBC strategy is performed on the existing coupled-gate LSTM to verify the analysis. Experimental results show that GBC achieves an additional <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$32\times $ </tex-math></inline-formula> (overall <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$42.7\times $ </tex-math></inline-formula> ) compression rate with negligible accuracy loss. Finally, hardware experiments conducted on Xilinx ADM-PCIE-7V3 FPGAs also demonstrate that the accelerator designed in this paper achieves an improvement of 7.4%-191.5% in energy efficiency compared to the state-of-the-art designs.

Gated Adaptive Hierarchical Attention Unit Neural Networks for the Life Prediction of Servo Motors

The remaining useful life (RUL) prediction of servo motors plays a greatly indispensable role in preventing abnormal and inefficient operation of servo drive system and even major safety accident. In this article, a novel variant of recurrent neural networks, called gated adaptive hierarchical attention unit (GAHAU), is constructed to efficiently predict the RULs of servo motors, which can not only pay attention to the outputs of reset and update gates but also adaptively divide the hierarchy of the sequence information in attention gate. The updating formulas of GAHAU are derived. With the raw vibration signals of servo motors, a quadratic function-based deep convolutional autoencoder is used for automatically generating the health indicator vector. Based on the constructed HI vector, the RULs of servo motors are accurately predicted by GAHAU. The experiments on servo motors demonstrate that the presented GAHAU has remarkable prediction performance especially for the long-term prediction. Compared to the existing typical RUL prediction methods, the proposed methodology has stronger predictive ability according to the comparative results.