Variant Of Recurrent Neural Network Research Articles

Deep Learning-Based Named Entity Recognition System Using Hybrid Embedding

Retrieval of meaningful information out of voluminous data available on the internet is a big challenge these days. Named entity recognition system deals with this challenge efficiently and achieves promising results in accessing information of interest in several NLP applications. The present study proposes a deep learning-based named entity recognition system using hybrid embedding which is the combination of fasttext and bidirectional LSTM based character embedding. These embeddings capture the contextual, syntactic and semantic properties of the text which improves the cognitive power of deep learning methods. We have performed different experiments with important variants of recurrent neural network (RNN) namely long short-term memory network (LSTM) and bidirectional LSTM as well as gated recurrent unit (GRU) and bidirectional GRU on manually annotated Punjabi dataset and annotated Hindi dataset collected from international joint conference on natural language processing (IJCNLP-08) website. We have also explored different word embeddings and character embeddings for named entity recognition task. Out of all the experiments, the bidirectional GRU model using hybrid embedding has outperformed with precision, recall, and f-score values of 84%, 83%, and 83.50% respectively for Punjabi named entity recognition and 75%, 77%, 75.99% respectively for Hindi named entity recognition.

Performance Evaluation of Recurrent Neural Networks Applied to Indoor Camera Localization

Researchers in robotics and computer vision are experimenting with the image-based localization of indoor cameras. Implementation of indoor camera localization problems using a Convolutional neural network (CNN) or Recurrent neural network (RNN) is more challenging from a large image dataset because of the internal structure of CNN or RNN. We can choose a preferable CNN or RNN variant based on the problem type and size of the dataset. CNN is the most flexible method for implementing indoor localization problems. Despite CNN's suitability for hyper-parameter selection, it requires a lot of training images to achieve high accuracy. In addition, overfitting leads to a decrease in accuracy. Introduce RNN, which accurately keeps input images in internal memory to solve these problems. Longshort-term memory (LSTM), Bi-directional LSTM (BiLSTM), and Gated recurrent unit (GRU) are three variants of RNN. We may choose the most appropriate RNN variation based on the problem type and dataset. In this study, we can recommend which variant is effective for training more speedily and which variant produces more accurate results. Vanishing gradient issues also affect RNNs, making it difficult to learn more data. Overcome the gradient vanishing problem by utilizing LSTM. The BiLSTM is an advanced version of the LSTM and is capable of higher performance than the LSTM. A more advanced RNN variant is GRU which is computationally more efficient than an LSTM. In this study, we explore a variety of recurring units for localizing indoor cameras. Our focus is on more powerful recurrent units like LSTM, BiLSTM, and GRU. Using the Microsoft 7-Scenes and InteriorNet datasets, we evaluate the performance of LSTM, BiLSTM, and GRU. Our experiment has shown that the BiLSTM is more efficient in accuracy than the LSTM and GRU. We also observed that the GRU is faster than LSTM and BiLSTM

Emotion Recognition from EEG Signals Using Recurrent Neural Networks

The application of electroencephalogram (EEG)-based emotion recognition (ER) to the brain–computer interface (BCI) has become increasingly popular over the past decade. Emotion recognition systems involve pre-processing and feature extraction, followed by classification. Deep learning has recently been used to classify emotions in BCI systems, and the results have been improved when compared to classic classification approaches. The main objective of this study is to classify the emotions from electroencephalogram signals using variant recurrent neural network architectures. Three architectures are used in this work for the recognition of emotions using EEG signals: RNN (recurrent neural network), LSTM (long short-term memory network), and GRU (gated recurrent unit). The efficiency of these networks, in terms of performance measures was confirmed by experimental data. The experiment was conducted by using the EEG Brain Wave Dataset: Feeling Emotions, and achieved an average accuracy of 95% for RNN, 97% for LSTM, and 96% for GRU for emotion detection problems.

IoT-Cloud-Based Smart Healthcare Monitoring System for Heart Disease Prediction via Deep Learning

The Internet of Things confers seamless connectivity between people and objects, and its confluence with the Cloud improves our lives. Predictive analytics in the medical domain can help turn a reactive healthcare strategy into a proactive one, with advanced artificial intelligence and machine learning approaches permeating the healthcare industry. As the subfield of ML, deep learning possesses the transformative potential for accurately analysing vast data at exceptional speeds, eliciting intelligent insights, and efficiently solving intricate issues. The accurate and timely prediction of diseases is crucial in ensuring preventive care alongside early intervention for people at risk. With the widespread adoption of electronic clinical records, creating prediction models with enhanced accuracy is key to harnessing recurrent neural network variants of deep learning possessing the ability to manage sequential time-series data. The proposed system acquires data from IoT devices, and the electronic clinical data stored on the cloud pertaining to patient history are subjected to predictive analytics. The smart healthcare system for monitoring and accurately predicting heart disease risk built around Bi-LSTM (bidirectional long short-term memory) showcases an accuracy of 98.86%, a precision of 98.9%, a sensitivity of 98.8%, a specificity of 98.89%, and an F-measure of 98.86%, which are much better than the existing smart heart disease prediction systems.

Reducing echo state network size with controllability matrices.

Echo state networks are a fast training variant of recurrent neural networks excelling at approximating nonlinear dynamical systems and time series prediction. These machine learning models act as nonlinear fading memory filters. While these models benefit from quick training and low complexity, computation demands from a large reservoir matrix are a bottleneck. Using control theory, a reduced size replacement reservoir matrix is found. Starting from a large, task-effective reservoir matrix, we form a controllability matrix whose rank indicates the active sub-manifold and candidate replacement reservoir size. Resulting time speed-ups and reduced memory usage come with minimal error increase to chaotic climate reconstruction or short term prediction. Experiments are performed on simple time series signals and the Lorenz-1963 and Mackey-Glass complex chaotic signals. Observing low error models shows variation of active rank and memory along a sequence of predictions.

Linking Transformer to Hawkes Process for Information Cascade Prediction (Student Abstract)

Information cascade is typically formalized as a process of (simplified) discrete sequence of events, and recent approaches have tackled its prediction via variants of recurrent neural networks. However, the information diffusion process is essentially an evolving directed acyclic graph (DAG) in the continuous-time domain. In this paper, we propose a transformer enhanced Hawkes process (Hawkesformer), which links the hierarchical attention mechanism with Hawkes process to model the arrival stream of discrete events continuously. A two-level attention architecture is used to parameterize the intensity function of Hawkesformer, which captures the long-term dependencies between nodes in graph and better embeds the cascade evolution rate for modeling short-term outbreaks. Experimental results demonstrate the significant improvements of Hawkesformer over the state-of-the-art.

Short-term multi-hour ahead country-wide wind power prediction for Germany using gated recurrent unit deep learning

In recent years, wind power has emerged as an important source of renewable energy. When onshore and offshore wind farm regions are connected to the grid for power generation, consistent multi-location short-term wind power predictions are extremely valuable in terms of assuring the power system's safety, sustainability, and economic operation. An abrupt variation in wind power generation influences the efficiency of the regional power grid. This makes accurate short-term forecasting essential for high-level planning and scheduling of power grids. To address the issue, this paper presents two variants of recurrent neural networks (RNN): gated recurrent unit (GRU) and long short-term memory (LSTM) models considering substantially better prediction accuracy to forecast a country-wide (Germany) wind power data for daily (t + 1), and multi-step (t + 3, t + 5, and t + 12) hours ahead. In addition, wind velocities [m/s] measured at heights of 2, 10, and 50-m (above ground level) are exploited as an essential characteristic among the available input variables and evaluated each feature subset based on four training divisions (80-20%, 70-30%, 60-40%, and 50-50%) and compared the results with ARIMA and SVR approaches in the literature. The findings reveal that the RNN-GRU model not only can achieve higher predicting accuracy but also has a faster learning speed over long sequences.

A two stream convolutional neural network with bi-directional GRU model to classify dynamic hand gesture

Dynamic hand gesture recognition is still an interesting topic for the computer vision community. A set of feature vectors can represent any hand gesture. A Recurrent Neural Network (RNN) can recognize these feature vectors as a hand gesture that analyzes the temporal and contextual information of the gesture sequence. Thus, we proposed a hybrid deep learning framework to recognize dynamic hand gestures. In the Hybrid model GoogleNet is pipelined with a Bidirectional GRU unit to recognize the dynamic hand gesture. Dynamic hand gestures consist of many frames, and features of each frame need to be extracted to get the temporal and dynamic information of the performed gesture. As RNN takes input as a sequence of feature vectors, we extract features from videos using pretrained GoogleNet. As Gated Recurrent Unit is one of the variants of RNN to classify the sequential data, we created a feature vector that corresponds to each video and passed it to the bidirectional GRU (BGRU) network to classify the gestures. We evaluate our model on four publicly available hand gesture datasets. The proposed method performs well and is comparable with the existing methods. For instance, we achieved 98.6% accuracy on Northwestern University Hand Gesture(NWUHG), 99.6% on SKIG, 99.4% on Cambridge Hand Gesture (CHG) datasets respectively. We performed our experiments on DHG14/28 dataset and achieved an accuracy of 97.8% with 14-gesture classes and 92.1% on 28-gesture classes. DHG14/28 dataset contains skeleton and depth data, and our proposed model used depth data and achieved comparable accuracy.

A mobile traffic load prediction based on recurrent neural network: A case of telecommunication in Afghanistan

This paper investigates the prediction of mobile traffic load based on four variants of recurrent neural networks, which are the simple long short-term memory (LSTM), stacked LSTM, gated recurrent unit (GRU) and bidirectional LSTM. In the considered schemes, the mobile traffic load of 15 min ahead of time is estimated based on the previous mobile traffic load data. The performance of the proposed scheme is verified using realistic traffic load data collected from the base station located in Kabul city, Afghanistan, which belongs to the SALAAM telecommunication operator during December 2020 and January 2021. Through performance evaluation, the authors confirm that the traffic load can be predicted with high accuracy using considered schemes and the GRU-based scheme outperforms other schemes in terms of accuracy.

Recurrent neural networks for short-term and long-term prediction of geothermal reservoirs

Accurate prediction of geothermal reservoir responses to alternative energy production scenarios is critical for optimizing the development of the underlying resources. While the conventional physics-based models offer a comprehensive prediction tool, data-driven models provide an efficient alternative to build fit-for-purpose predictive models by extracting and using the statistical patterns in the collected data to make predictions. The recurrent neural network (RNN) is a data-driven model that is commonly applied to predict time series sequences. This paper presents a variant of RNN that also utilizes the efficiency of convolutional neural networks (CNN) for the prediction of energy production from geothermal reservoirs. Specifically, a CNN–RNN architecture is developed that takes historical well controls as input (features) and their corresponding production response data as output (labels) to learn an input-output mapping that can predict the future well production responses/performance for any given future well control inputs. The model is paired with a labeling scheme to handle real field disturbances that create data gaps. In addition to the model structure, we introduce a thorough workflow for applying the model, which includes data pre-processing, feature selection, as well as different training strategies for short-term and long-term prediction. The performance and accuracy of the model are evaluated by applying it to multiple datasets, including a field reservoir model.

Long short-term enhanced memory for sequential recommendation

Sequential recommendation is a stream of studies on recommender systems, which focuses on predicting the next item a user interacts with by modeling the dynamic sequence of user-item interactions. Since being born to explore the dynamic tendency of variable-length temporal sequence, Recurrent Neural Networks (RNNs) have been paid much attention in this area. However, the inherent defects caused by the network structure of RNNs have limited their applications in sequential recommendation, which are mainly shown on two factors: RNNs tend to make point-wise predictions and ignore the collective dependencies because the temporal relationships between items change monotonically; RNNs are likely to forget the essential information during processing long sequences. To solve these problems, researchers have done much work to enhance the memory mechanism of RNNs. However, although previous RNN-based methods have achieved promising performance by taking advantage of external knowledge with other advanced techniques, the improvement of the intrinsic property of existing RNNs has not been explored, which is still challenging. Therefore, in this work, we propose a novel architecture based on Long Short-Term Memories (LSTMs), a broadly-used variant of RNNs, specific for sequential recommendation, called Long Short-Term enhanced Memory (LSTeM), which boosts the memory mechanism of original LSTMs in two ways. Firstly, we design a new structure of gates in LSTMs by introducing a “Q-K-V” triplet, a mechanism to accurately and properly model the correlation between the current item and the user’s historical behaviors at each time step. Secondly, we propose a “recover gate” to remedy the inadequacy of memory caused by the forgetting mechanism, which works with a dynamic global memory embedding. Extensive experiments have demonstrated that LSTeM achieves comparable performance to the state-of-the-art methods on the challenging datasets for sequential recommendation.

Recurrent neural network variants based model for Cassini-Huygens spacecraft trajectory modifications recognition

Over the last 13.7 years period of the Cassini mission, amendments to the spacecraft’s flight path were needed. This research is being carried out as there is a limited number of studies that use a temporal discrimination analysis to handle raw data. More complex inspection and analysis of the collected broad trajectory dataset is necessary to classify orbital events in the signal travel period (approximately 88 minutes on the Earth-Cassini travel channel length). This paper provides an innovative, in-depth learning method to identify offline modifications in the Cassini spacecraft trajectory. The models are based on variants of Recurrent Neural Networks (RNNs: Gated Recurrent Unit (GRU)/ Long Short-Term Memory (LSTM)/ Bidirectional Long Short-Term Memory (BiLSTM)) to derive valuable data and learn the inner data structure of the time sequence, along with the penetration of long-term and short-term phase-dependencies of the RNNs layers. To validate our models, we used a variety of statistical approaches in our analysis. A considerable number of tests have been carried out, and the findings obtained have shown that the GRU and LSTM give a substantial boost to increasing the efficiency of the detection mechanism. The proposed model would consolidate potential exploration in outer space exploration to accommodate massive databases, search for correlations, and recognize complex events and outliers with an accuracy that exceeds 99 %. This method can be utilized for similar detection processes within the future outer space expedition. The results show that binary classifications of Matthews Correlation Coefficient (MCC) are more accurate than F_1 score.

Goal-Directed Planning and Goal Understanding by Extended Active Inference: Evaluation through Simulated and Physical Robot Experiments.

We show that goal-directed action planning and generation in a teleological framework can be formulated by extending the active inference framework. The proposed model, which is built on a variational recurrent neural network model, is characterized by three essential features. These are that (1) goals can be specified for both static sensory states, e.g., for goal images to be reached and dynamic processes, e.g., for moving around an object, (2) the model cannot only generate goal-directed action plans, but can also understand goals through sensory observation, and (3) the model generates future action plans for given goals based on the best estimate of the current state, inferred from past sensory observations. The proposed model is evaluated by conducting experiments on a simulated mobile agent as well as on a real humanoid robot performing object manipulation.

CatVRNN: Generating category texts via multi-task learning

Controlling the model to generate texts of different categories is a challenging task that is receiving increasing attention. Recently, generative adversarial networks (GANs) have shown promising results for category text generation. However, the texts generated by GANs usually suffer from problems of mode collapse and training instability. To avoid the above problems, in this study, inspired by multi-task learning, a novel model called category-aware variational recurrent neural network (CatVRNN) is proposed. In this model, generation and classification tasks are trained simultaneously to generate texts of different categories. The use of multi-task learning can improve the quality of the generated texts, when the classification task is appropriate. In addition, a function is proposed to initialize the hidden state of the CatVRNN to force the model to generate texts of a specific category. Experimental results on three datasets demonstrate that the model can outperform state-of-the-art text generation methods based on GAN in terms of diversity of generated texts.

Comparative study of a mathematical epidemic model, statistical modeling, and deep learning for COVID-19 forecasting and management

The COVID-19 pandemic has caused a global crisis with 47,209,305 confirmed cases and 1,209,505 confirmed deaths worldwide as of November 2, 2020. Forecasting confirmed cases and understanding the virus dynamics is necessary to provide valuable insights into the growth of the outbreak and facilitate policy-making regarding virus containment and utilization of medical resources. In this study, we applied a mathematical epidemic model (MEM), statistical model, and recurrent neural network (RNN) variants to forecast the cumulative confirmed cases. We proposed a reproducible framework for RNN variants that addressed the stochastic nature of RNN variants leveraging z-score outlier detection. We incorporated heterogeneity in susceptibility into the MEM considering lockdowns and the dynamic dependency of the transmission and identification rates which were estimated using Poisson likelihood fitting. While the experimental results demonstrated the superiority of RNN variants in forecasting accuracy, the MEM presented comprehensive insights into the virus spread and potential control strategies.

Graph Substructure Assembling Network with Soft Sequence and Context Attention

There has been a surge of researchers' interest in building predictive models over graphs. However, the overwhelming complexity of graph space often makes it challenging to extract interpretable and discriminative structural features for graph classification. In this work, we propose a new graph neural network model called Substructure Assembling Network (SAN) to learn graph representations for classification. The key innovation is a unified Substructure Assembling Unit (SAU), which is a variant of Recurrent Neural Network (RNN) designed to hierarchically assemble useful pieces of graph components so as to fabricate discriminative substructures. A key challenge is that SAUs need to process the neighbors of a node sequentially while no natural order is defined therein. SAN tries to make the model insensitive to neighborhood orders by randomly shuffling neighborhood sequences in training. However, this could suffer high variance, especially when the neighborhood size is large. Hence, we further propose to equip SAN with a novel module named Soft Sequence with Context Attention (SSCA). SAN-SSCA employs the proposed context attention technique to learn the best "soft" permutation of the neighbors w.r.t. classification. It helps the model achieve higher accuracy as well as lower variance. Experiments confirm the effectiveness of SAN-SSCA.

CNN-LSTM based classification of polo like kinase family of Proteins: An emerging cancer drug target

Artificial intelligence and deep learning are becoming an inevitable part of our life. Deep learning models are giving contributions to the identification of genetic causes behind various diseases affecting the human community. The prognosis and diagnosis of such diseases can be recommended with the help of artificial intelligence. Here we are proposing a novel deep learning model that employs the legendary deep learning architectures such as convolutional neural network and the variant of recurrent neural network known as long short-term memory network to classify the protein sequences belonging to the five-member polo like kinase family, a subclass of Serine-Threonine kinases, which is considered as an active anti-cancer drug target these days. The proposed deep learning model was trained and tested on sequences collected from biological sequence databases and classified the new sequences to their corresponding classes with an accuracy of 97.6%. Furthermore, this model could untangle the efforts associated with sequence annotation and classification, which is always a tedious and exigent task.

Refractive Index Perception and Prediction of Radio wave through Recursive Neural Networks using Meteorological Data Parameters

Radio refractivity is very crucial in the optimal performance of radio systems and is one of the attributes that affect electromagnetic waves in the troposphere. This study presented a comparison of different variants of recurrent neural networks to predict radio refractivity index. The radio refractivity index is predicted based on forty-one years (1980 to 2020) metrological data obtained from the MERRA-2 data re-analysis database. The refractivity index was computed using International Telecommunication Union (ITU) standard. The correlation refractivity index was categorized into strong, weak and no correlation. Rainfall, relative humidity, and air pressure fall in the first category, the temperature falls in the second category while wind speed falls in the last one. The true future and predicted values of the radio refractivity index are close with GRU performing better than the other two models (LSTM and BiLSTM) which proves the accuracy of the proposed model. In conclusion, the proposed model can establish a radio refractivity status of locations at different times of the season, which is of great importance in the effective design, development, and deployment of radio communication systems.

Cyclone Trajectory and Intensity Prediction with Uncertainty Quantification Using Variational Recurrent Neural Networks

Cyclone Trajectory and Intensity Prediction with Uncertainty Quantification Using Variational Recurrent Neural Networks

Learning Generative RNN-ODE for Collaborative Time-Series and Event Sequence Forecasting

Time-series and event sequences are widely collected data types in real-world applications. Modeling and forecasting of such temporal data play an important role in an informed decision-making process. A major limitation of previous methods is that they either focus on time-series or events, rather than the combination of the two worlds. In fact, the two types of data often provide complementary information, emphasizing the necessity of jointly modeling the both. In this paper, we propose the RNN-ODE collaborative model for joint modeling and forecasting of heterogeneous time-series and event sequence data, which combines several useful techniques from both Bayesian and deep learning for its interpretability. Specifically, we devise a tailored encoder to combine the advances in deep temporal point processes models and variational recurrent neural networks. To predict the probability of event occurrence over an arbitrary continuous-time horizon, we base our model on the mathematical foundation of Neural Ordinary Differential Equations (NODE). Extensive experimental results on simulations and real data sets show that compared with existing methods, our integrated approach can achieve more competitive forecasting performance of both time-series and event sequences.