Temporal Recurrent Neural Network Research Articles

Spatial-Temporal Preserving Multimodal Algorithm for EEG-based Emotion Recognition

Emotion recognition has been the focus of some human computer interaction researchers. Designing an efficient algorithm for emotion recognition is crucial for its potential use in HCI. Many algorithms have been developed for emotion recognition and a majority of them is based on electroencephalography (EEG). Despite the success of spatial–temporal recurrent neural network (STRNN) and multimodal deep neural networks, they can still be improved. This paper proposes an algorithm which tries to combine STRNN with multimodal deep neural network, where EEG signals are classified using STRNN and eye movement signals are classified using SVM. The results from the two classifiers are fused by max fusion. This paper shows that by taking spatial and temporal information of EEG signals into consideration and using multimodal fusion, the performance of emotion recognition can be improved.

Spatial–temporal recurrent reinforcement learning for autonomous ships

This paper proposes a spatial–temporal recurrent neural network architecture for deep Q-networks that can be used to steer an autonomous ship. The network design makes it possible to handle an arbitrary number of surrounding target ships while offering robustness to partial observability. Furthermore, a state-of-the-art collision risk metric is proposed to enable an easier assessment of different situations by the agent. The COLREG rules of maritime traffic are explicitly considered in the design of the reward function. The final policy is validated on a custom set of newly created single-ship encounters called ‘Around the Clock’ problems and the commonly used Imazu (1987) problems, which include 18 multi-ship scenarios. Performance comparisons with artificial potential field and velocity obstacle methods demonstrate the potential of the proposed approach for maritime path planning. Furthermore, the new architecture exhibits robustness when it is deployed in multi-agent scenarios and it is compatible with other deep reinforcement learning algorithms, including actor-critic frameworks.

Performance analysis of sentiment classification based neural network

Deep learning has more significant advantages for word embedding technology than sentiment analysis. This paper studies the application of deep learning on the word embedding problem in context, mainly discusses the RNN model with Word2Vec and without Word2Vec, then compares and analyzes their performance in the experiment, mainly evaluating the accuracy and test loss of seven models. This paper compares and illustrates the model which gets the different results in experiments, complementing the model and re-running the model, and analyzing the reasons for the difference in the performance of each model. The seven models are a single-layer neural network, multiple-layer (two and three) feed-forward neural networks, Convolutional Neural Network (CNN)- A feedforward neural network, which consists of single or multiple convolutional layers, pooling layers, and a fully connected layer on top, so this model is good at image processing. Long Short Term Memory (LSTM)- A temporal recurrent neural network, the advantage of the model is it could solve the gradient disappearance and explosion problem when it handles the long-sequence problem. Bi-directional Long Short Term Memory (Bi-LSTM)-Composed of forwarding LSTM and backward LSTM, it is very common for sequence labelling tasks that are related to the top and bottom, which are often used to model context information in NLP. Bi-directional Encoder Representation from Transformers (BERT)- A bidirectional language model. Finally, this paper analyses and evaluates these models with a specific illustration and research.

Involuntary Table Tennis Recognition using Recurrent Neural Network on The Perception-Based Data

Sport performance analysis and sports practise are inextricably linked. It is critical to assist coaches in analysing and improving the performance of their athletes throughout training or game sessions. Because of technological advancements, notational analysis of video footage using multiple software packages is now possible. Unfortunately, the coach must manually recognise the acts before proceeding with additional analysis. The goal of this research is to create an automated system for recognising Table Tennis shots in widely available broadcasted videos using a pre-trained Recurrent Neural Network (RNN) approach. We provide a novel method for gathering video data from table tennis games and performing stroke detection and classification. Using the proposed setup, a diversified dataset encompassing video data of 4 basic strokes taken from 4 professional table tennis players, totalling 2000 films, was collected. With a validation accuracy of 97.02%, the temporal recurrent neural network model is created. Furthermore, the neural network generalises well over the data of a player who was not included in the training and validation datasets, categorising new strokes with an overall best accuracy of 91.12%. Several model architectures were trained for stroke recognition using machine learning and deep learning methodologies, and their results were compared and benchmarked. The model's inferences, such as performance monitoring and stroke comparison, have been examined. As a result, we are contributing to the creation of a computer vision-based sports analytics system for table tennis, namely a player's strokes, and is particularly insightful for performance enhancement.

MFGNet: Dynamic Modality-Aware Filter Generation for RGB-T Tracking

Many RGB-T trackers attempt to attain robust feature representation by utilizing an adaptive weighting scheme (or attention mechanism). Different from these works, we propose a new dynamic modality-aware filter generation module (named MFGNet) to boost the message communication between visible and thermal data by adaptively adjusting the convolutional kernels for various input images in practical tracking. Given the image pairs as input, we first encode their features with the backbone network. Then, we concatenate these feature maps and generate dynamic modality-aware filters with two independent networks. The visible and thermal filters will be used to conduct a dynamic convolutional operation on their corresponding input feature maps respectively. Inspired by residual connection, both the generated visible and thermal feature maps will be summarized with input feature maps. The augmented feature maps will be fed into the RoI align module to generate instance-level features for subsequent classification. To address issues caused by heavy occlusion, fast motion and out-of-view, we propose to conduct a joint local and global search by exploiting a new direction-aware target driven attention mechanism. The spatial and temporal recurrent neural network is used to capture the direction-aware context for accurate global attention prediction. Extensive experiments on three large-scale RGB-T tracking benchmark datasets validated the effectiveness of our proposed algorithm.

SPRNN: A spatial–temporal recurrent neural network for crowd flow prediction

The capability of predicting the future trends of crowds has rendered crowd flow prediction more critical in building intelligent transportation systems, and attracted substantial research efforts. The trend of crowd flows is closely related to time and the urban topography. Therefore, extracting and leveraging both spatial features and temporal features are key gradients for effectively predicting crowd flows. Many previous works extract spatial features from crowd-flow data in an iteration way. As a result, models suffer from a heavy computation cost while ignoring details of road topology and structure information. Meanwhile, temporal features, including short-term features and long-term features, are separately extracted. The fusion of all features at the last stage before accomplishing the prediction also neglects the underlying associativity between various features. To address the limitations, we leverage spatial features by extracting structural information of road structures, such as road connection, road density, road width, etc. Rather than extracting spatial features from crowd-flow data, we capture them from images of city maps by adopting convolutional neural networks. Moreover, we implement a new sequence feature fusion mechanism to merge both spatial features and temporal features from various time scales so as to predict crowd flows. We conduct extensive experiments to evaluate our model on three benchmark datasets. The experimental results demonstrate that the model outperforms 15 state-of-the-art methods. The source code is available at: https://github.com/CVisionProcessing/SPRNN.

MST-RNN: A Multi-Dimension Spatiotemporal Recurrent Neural Networks for Recommending the Next Point of Interest

With the increasing popularity of location-aware Internet-of-Vehicle services, the next-Point-of-Interest (POI) recommendation has gained significant research interest, predicting where drivers will go next from their sequential movements. Many researchers have focused on this problem and proposed solutions. Machine learning-based methods (matrix factorization, Markov chain, and factorizing personalized Markov chain) focus on a POI sequential transition. However, they do not recommend the user’s position for the next few hours. Neural network-based methods can model user mobility behavior by learning the representations of the sequence data in the high-dimensional space. However, they just consider the influence from the spatiotemporal dimension and ignore many important influences, such as duration time at a POI (Point of Interest) and the semantic tags of the POIs. In this paper, we propose a novel method called multi-dimension spatial–temporal recurrent neural networks (MST-RNN), which extends the ST-RNN and exploits the duration time dimension and semantic tag dimension of POIs in each layer of neural networks. Experiments on real-world vehicle movement data show that the proposed MST-RNN is effective and clearly outperforms the state-of-the-art methods.

Wi-Exercise: An Indoor Human Movement Detection Method Based on Bidirectional LSTM Attention

Human motion recognition has become a hot topic in the field of human-computer interaction. Due to the traditional method of detecting human movement in indoor environments, there are still problems such as high hardware costs, influenced by the environment and the need for experimenters to be equipped with relevant equipment. With the large-scale deployment of wireless devices and the establishment of wireless network infrastructure, indoor personnel movement recognition systems based on wireless signals are gradually proposed. In this paper, we propose a device-independent motion-detection method Wi-exercise based on temporal recurrent neural networks. The acquired channel state information (CSI) is preprocessed by wavelet transform, and the output CSI matrix is filtered by the principal component analysis (PCA) to extract the feature values. The wavelet function is processed to further remove the noise also to make the action data more distinctive, and then PCA is used to track the entire CSI time series to improve and maintain its dimensional energy. The preprocessed data are subjected to feature extraction to obtain CSI of human motion and establish feature sequences, and then the action model is trained using the bidirectional long short-term memory attention (ABLSTM) mechanism to finally realize human motion recognition. Finally, the robustness of Wi-exercise is tested experimentally, and its performance is compared with that of existing recognition methods. The experimental results show that the average recognition rate of the method is 94.43% in the complex indoor environment, and the method is better than the traditional classifier method for action recognition, with lower and higher recognition accuracy.

A novel IoT-fog-cloud-based healthcare system for monitoring and predicting COVID-19 outspread.

Rapid communication of viral sicknesses is an arising public medical issue across the globe. Out of these, COVID-19 is viewed as the most critical and novel infection nowadays. The current investigation gives an effective framework for the monitoring and prediction of COVID-19 virus infection (C-19VI). To the best of our knowledge, no research work is focused on incorporating IoT technology for C-19 outspread over spatial–temporal patterns. Moreover, limited work has been done in the direction of prediction of C-19 in humans for controlling the spread of COVID-19. The proposed framework includes a four-level architecture for the expectation and avoidance of COVID-19 contamination. The presented model comprises COVID-19 Data Collection (C-19DC) level, COVID-19 Information Classification (C-19IC) level, COVID-19-Mining and Extraction (C-19ME) level, and COVID-19 Prediction and Decision Modeling (C-19PDM) level. Specifically, the presented model is used to empower a person/community to intermittently screen COVID-19 Fever Measure (C-19FM) and forecast it so that proactive measures are taken in advance. Additionally, for prescient purposes, the probabilistic examination of C-19VI is quantified as degree of membership, which is cumulatively characterized as a COVID-19 Fever Measure (C-19FM). Moreover, the prediction is realized utilizing the temporal recurrent neural network. Additionally, based on the self-organized mapping technique, the presence of C-19VI is determined over a geographical area. Simulation is performed over four challenging datasets. In contrast to other strategies, altogether improved outcomes in terms of classification efficiency, prediction viability, and reliability were registered for the introduced model.

A wavelet-based multi-dimensional temporal recurrent neural network for stencil printing performance prediction

The solder paste printing (SPP) is a critical procedure in a surface mount technology (SMT) based assembly line, which is one of the major attributes to the defect of the printed circuit boards (PCBs). The quality of SPP is influenced by multiple factors, such as the squeegee speed, pressure, the stencil separation speed, cleaning frequency, and cleaning profile. During printing, the printer environment is dynamically varying due to the physical change of solder paste, which can result in a dynamic variation of the relationships between the printing results and the influential factors. To reduce the printing defects, it is critical to understand such dynamic relationships. This research focuses on determining the printing performance during printing by implementing a wavelet filtering-based temporal recurrent neural network. To reduce the noise factor in the solder paste inspection (SPI) data, this research applies a three-dimensional dual-tree complex wavelet transformation for low-pass noise filtering and signal reconstruction. A recurrent neural network is utilized to model the performance prediction with low noise interference. Both printing sequence and process setting information are considered in the proposed recurrent network model. The proposed approach is validated using practical dataset and compared with other commonly used data mining approaches. The results show that the proposed wavelet-based multi-dimensional temporal recurrent neural network can effectively predict the printing process performance and can be a high potential approach in reducing the defects and controlling cleaning frequency. The proposed model is expected to advance the current research in the application of smart manufacturing in surface mount technology.

Residue buildup predictive modeling for stencil cleaning profile decision-making using recurrent neural network

This research proposes a novel framework to control the stencil cleaning profile selection in the stencil printing process (SPP). The SPP is a major contributor to yield loss in surface mount technology (SMT). Enhancement in SPP performance is critical to improving the printed circuit board (PCB) assembly line. The selection of a solvent-based or a dry-based cleaning profile is challenging, but the choice determines the effectiveness and efficiency of the stencil cleaning operation. The amount of residue buildup under the stencil is the main criterion used to decide the appropriate cleaning profile in SPP. In this research, a multi-dimensional temporal recurrent neural network (RNN) approach is used to accurately predict the amount of residue buildup on the underneath surface of the stencil in real-time. Specifically, the long short-term memory (LSTM) architecture is trained using actual residue buildup data. The proposed LSTM prediction model is compared with other state-of-the-art regression models such as multilayer perceptron (MLP) and ensemble learning models. Experimental results show the proposed LSTM model outperforms the state-of-the-art regression models and accurately predicts the stencil status. The proposed research aids decision-makers in the SPP line to select the appropriate stencil cleaning profile adaptively and in real-time. As a result, the overall SPP performance is improved.

Learning deep hierarchical and temporal recurrent neural networks with residual learning

Learning both hierarchical and temporal dependencies can be crucial for recurrent neural networks (RNNs) to deeply understand sequences. To this end, a unified RNN framework is required that can ease the learning of both the deep hierarchical and temporal structures by allowing gradients to propagate back from both ends without being vanished. The residual learning (RL) has appeared as an effective and less-costly method to facilitate backward propagation of gradients. The significance of the RL is exclusively shown for learning deep hierarchical representations and temporal dependencies. Nevertheless, there is lack of efforts to unify these finding into a single framework for learning deep RNNs. In this study, we aim to prove that approximating identity mapping is crucial for optimizing both hierarchical and temporal structures. We propose a framework called hierarchical and temporal residual RNNs, to learn RNNs by approximating identity mappings across hierarchical and temporal structures. To validate the proposed method, we explore the efficacy of employing shortcut connections for training deep RNNs structures for sequence learning problems. Experiments are performed on Penn Treebank, Hutter Prize and IAM-OnDB datasets and results demonstrate the utility of the framework in terms of accuracy and computational complexity. We demonstrate that even for large datasets exploiting parameters for increasing network depth can gain computational benefits with reduced size of the RNN "state".

Learning Time Series Associated Event Sequences With Recurrent Point Process Networks.

Real-world sequential data are often generated based on complicated and latent mechanisms, which can be formulated as event sequences occurring in the continuous time domain. In addition, continuous signals may often be associated with event sequences and be formulated as time series with fixed time lags. Traditionally, event sequences are often modeled by parametric temporal point processes, which use explicitly defined conditional intensity functions to quantify the occurrence rates of events. However, these parametric models often merely take one-side information from event sequences into account while ignoring the information from concurrent time series, and their intensity functions are usually designed for specific tasks dependent on prior knowledge. To tackle the above-mentioned problems, we propose a model called recurrent point process networks which instantiates temporal point process models with temporal recurrent neural networks (RNNs). In particular, the intensity functions of the proposed model are modeled by two RNNs: one temporal RNN capturing the relationships among events and the other RNN updating intensity functions based on time series. Furthermore, an attention mechanism is introduced, which uncovers influence strengths among events with good interpretability. Focusing on challenging tasks such as temporal event prediction and underlying relational network mining, we demonstrate the superiority of our model on both synthetic and real-world data.

Improving EEG-Based Motor Imagery Classification via Spatial and Temporal Recurrent Neural Networks.

Motor imagery (MI) based Brain-Computer Interface (BCI) is an important active BCI paradigm for recognizing movement intention of severely disabled persons. There are extensive studies about MI-based intention recognition, most of which heavily rely on staged handcrafted EEG feature extraction and classifier design. For end-to-end deep learning methods, researchers encode spatial information with convolution neural networks (CNNs) from raw EEG data. Compared with CNNs, recurrent neural networks (RNNs) allow for long-range lateral interactions between features. In this paper, we proposed a pure RNNs-based parallel method for encoding spatial and temporal sequential raw data with bidirectional Long Short- Term Memory (bi-LSTM) and standard LSTM, respectively. Firstly, we rearranged the index of EEG electrodes considering their spatial location relationship. Secondly, we applied sliding window method over raw EEG data to obtain more samples and split them into training and testing sets according to their original trial index. Thirdly, we utilized the samples and their transposed matrix as input to the proposed pure RNNs- based parallel method, which encodes spatial and temporal information simultaneously. Finally, the proposed method was evaluated in the public MI-based eegmmidb dataset and compared with the other three methods (CSP+LDA, FBCSP+LDA, and CNN-RNN method). The experiment results demonstrated the superior performance of our proposed pure RNNs-based parallel method. In the multi-class trial-wise movement intention classification scenario, our approach obtained an average accuracy of 68.20% and significantly outperformed other three methods with an 8.25% improvement of relative accuracy on average, which proves the feasibility of our approach for the real-world BCI system.

Predicting the Next Location: A Recurrent Model with Spatial and Temporal Contexts

Spatial and temporal contextual information plays a key role for analyzing user behaviors, and is helpful for predicting where he or she will go next. With the growing ability of collecting information, more and more temporal and spatial contextual information is collected in systems, and the location prediction problem becomes crucial and feasible. Some works have been proposed to address this problem, but they all have their limitations. Factorizing Personalized Markov Chain (FPMC) is constructed based on a strong independence assumption among different factors, which limits its performance. Tensor Factorization (TF) faces the cold start problem in predicting future actions. Recurrent Neural Networks (RNN) model shows promising performance comparing with PFMC and TF, but all these methods have problem in modeling continuous time interval and geographical distance. In this paper, we extend RNN and propose a novel method called Spatial Temporal Recurrent Neural Networks (ST-RNN). ST-RNN can model local temporal and spatial contexts in each layer with time-specific transition matrices for different time intervals and distance-specific transition matrices for different geographical distances. Experimental results show that the proposed ST-RNN model yields significant improvements over the competitive compared methods on two typical datasets, i.e., Global Terrorism Database (GTD) and Gowalla dataset.

Non-linear autoregressive modelling by Temporal Recurrent Neural Networks for the prediction of freshwater phytoplankton dynamics

This study was aimed at developing a Temporal Autoregressive Recurrent Neural Network (TARNN) model that could predict time-series changes of phytoplankton dynamics in a regulated river ecosystem in South Korea. In recent years, a large quantity of ecological data has been globally accumulated in habitats monitored by Long-Term Ecological Research (LTER), and that data enabled ecologists to apply non-linear data-driven ecological modelling algorithms to their systems. Numerous studies have reported that empirical modelling algorithms such as Artificial Neural Networks (ANNs) were superior to conventional models in applicability, especially for systems where underlying ecological relationship was not fully understood. However, in order to process sufficient information from the target ecosystem or entity, the size of empirical models tended to become larger by applying diverse state variables or forcing functions to the models. Despite the fact that the developed models empirically had accurate performance in prediction or classification of target data, they are occasionally not free from complex structures (e.g., large number of input variables). The TARNN algorithm used in this study can be an alternative solution to overcome the increasing size and structural complexity of the models, based on model performance, which is hardly found in freshwater ecology. In this study, the performance of TARNN for freshwater ecological data was evaluated by being applied to two types of data sets of phytoplankton dynamics. The first data encompassed intensive seasonality (monthly averaged biovolume of Stephanodiscus hantzschii) which proliferated and dominated the algal assemblage (ca. 90%) annually in the lower Nakdong River during winters. The other data was daily sampled phytoplankton biomass (chlorophyll a) in which significant seasonality did not reside. The predictability of two TARNN models was compared with appropriate statistical models, i.e., data with seasonality was modelled by Seasonal Auto-Regressive Integrated Moving Average (SARIMA), and the other data with Simple Exponential Smoothing. In both cases of data sets, the TARNN models outperformed the statistical methods in accuracy of prediction (TARNN, r 2 > 0.9 for both seasonal and non-seasonal testing data sets; both statistical models, r 2 < 0.3 for the testing data sets). Especially, the capacity of prediction using the TARNN algorithm was not decreased by the absence of seasonality. Through the presented results, it can be assumed that TARNN can process non-linear autoregressive analysis successfully for ecological data. The development of non-linear autoregressive models for phytoplankton prediction can suggest more efficient direction for water quality management approaches of freshwater systems with long-term database. A real-time forecasting system for algal proliferation, one of the state-of-the-art predicting systems, can be identified using the linkage between the modelling technique and telemetry.

Identification of prosodic attitudes by a temporal recurrent network

Human speakers modulate the fundamental frequency (F0) of their utterances in order to express different ‘prosodic’ attitudes such as surprise or curiosity. How are these prosodic attitudes then decoded? The current research addresses the issue of how the temporal structure of F0 can be used in order to discriminate between prosodic attitudes in natural language using a temporal recurrent neural network (TRN) that was initially developed to simulate the neurophysiology of the primate frontostriatal system. In the TRN, a recurrent network of leaky integrator neurons encodes a continuous trajectory of internal states that characterizes the input sequence. The input to the model is a population coding of the continuous, time-varying values of the fundamental frequency (F0) of natural language sentences. We expose the model to an experiment based on one in which human subjects were required to discriminate between different prosodic attitudes (surprise, exclamation, question, etc.). After training, the model discriminates between six prosodic attitudes in new sentences at 82.52% correct, compared to 72.8% correct for human subjects. These results reveal (1) that F0 provides relevant information for prosodic attitude discrimination, and (2) that the TRN demonstrates a categorical sensitivity to this information that can be used for classifying new sentences.