Modeling Sequential Data Research Articles

Grow and Prune Compact, Fast, and Accurate LSTMs

Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-layer LSTM (H-LSTM) that adds hidden layers to LSTM's original one-level nonlinear control gates. H-LSTM increases accuracy while employing fewer external stacked layers, thus reducing the number of parameters and run-time latency significantly. We employ grow-and-prune (GP) training to iteratively adjust the hidden layers through gradient-based growth and magnitude-based pruning of connections. This learns both the weights and the compact architecture of H-LSTM control gates. We have GP-trained H-LSTMs for image captioning, speech recognition, and neural machine translation applications. For the NeuralTalk architecture on the MSCOCO dataset, our three models reduce the number of parameters by 38.7× [floating-point operations (FLOPs) by 45.5×], run-time latency by 4.5×, and improve the CIDEr-D score by 2.8 percent, respectively. For the DeepSpeech2 architecture on the AN4 dataset, the first model we generated reduces the number of parameters by 19.4× and run-time latency by 37.4 percent. The second model reduces the word error rate (WER) from 12.9 to 8.7 percent. For the encoder-decoder sequence-to-sequence network on the IWSLT 2014 German-English dataset, the first model we generated reduces the number of parameters by 10.8× and run-time latency by 14.2 percent. The second model increases the BLEU score from 30.02 to 30.98. Thus, GP-trained H-LSTMs can be seen to be compact, fast, and accurate.

AdapNet: Adaptability Decomposing Encoder-Decoder Network for Weakly Supervised Action Recognition and Localization.

The point process is a solid framework to model sequential data, such as videos, by exploring the underlying relevance. As a challenging problem for high-level video understanding, weakly supervised action recognition and localization in untrimmed videos have attracted intensive research attention. Knowledge transfer by leveraging the publicly available trimmed videos as external guidance is a promising attempt to make up for the coarse-grained video-level annotation and improve the generalization performance. However, unconstrained knowledge transfer may bring about irrelevant noise and jeopardize the learning model. This article proposes a novel adaptability decomposing encoder-decoder network to transfer reliable knowledge between the trimmed and untrimmed videos for action recognition and localization by bidirectional point process modeling, given only video-level annotations. By decomposing the original features into the domain-adaptable and domain-specific ones based on their adaptability, trimmed-untrimmed knowledge transfer can be safely confined within a more coherent subspace. An encoder-decoder-based structure is carefully designed and jointly optimized to facilitate effective action classification and temporal localization. Extensive experiments are conducted on two benchmark data sets (i.e., THUMOS14 and ActivityNet1.3), and the experimental results clearly corroborate the efficacy of our method.

Supervised Brain Network Learning Based on Deep Recurrent Neural Networks

Task-based functional magnetic resonance imaging (tfMRI) is a widely used neuroimaging technique in exploring brain networks and functions associated with cognitive behaviors. Traditionally, the general linear model (GLM) is the most popular method in tfMRI data analysis due to its simpleness and robustness. This model-driven method adopts a canonical hemodynamic response function (HRF) and its various derivatives to construct regressors in the design matrix and estimate changes in the tfMRI data. However, a possible limitation of current model-driven methods is that the HRF is fixed and non-adaptive which may overlook other diverse and concurrent brain networks. In order to overcome these limitations, we proposed a novel hybrid framework, supervised brain network learning based on deep recurrent neural networks (SUDRNN), to reconstruct the diverse and concurrent functional brain networks. Specifically, this hybrid framework first takes advantage of the great capacity of deep recurrent neural networks (DRNN) in modeling sequential data to learn the diverse regressors from real tfMRI data. After that, it utilizes the effective supervised dictionary learning (SDL) method to reconstruct both the task-related functional brain networks and other latent brain networks simultaneously. Extensive experiment results on different tfMRI datasets from Human connectome project (HCP) demonstrated the superiority of the proposed framework.

Hierarchical Multi-Granularity Attention- Based Hybrid Neural Network for Text Classification

Neural network-based approaches have become the driven forces for Natural Language Processing (NLP) tasks. Conventionally, there are two mainstream neural architectures for NLP tasks: the recurrent neural network (RNN) and the convolution neural network (ConvNet). RNNs are good at modeling long-term dependencies over input texts, but preclude parallel computation. ConvNets do not have memory capability and it has to model sequential data as un-ordered features. Therefore, ConvNets fail to learn sequential dependencies over the input texts, but it is able to carry out high-efficient parallel computation. As each neural architecture, such as RNN and ConvNets, has its own pro and con, integration of different architectures is assumed to be able to enrich the semantic representation of texts, thus enhance the performance of NLP tasks. However, few investigation explores the reconciliation of these seemingly incompatible architectures. To address this issue, we propose a hybrid architecture based on a novel hierarchical multi-granularity attention mechanism, named Multi-granularity Attention-based Hybrid Neural Network (MahNN). The attention mechanism is to assign different weights to different parts of the input sequence to increase the computation efficiency and performance of neural models. In MahNN, two types of attentions are introduced: the syntactical attention and the semantical attention. The syntactical attention computes the importance of the syntactic elements (such as words or sentence) at the lower symbolic level and the semantical attention is used to compute the importance of the embedded space dimension corresponding to the upper latent semantics. We adopt the text classification as an exemplifying way to illustrate the ability of MahNN to understand texts. The experimental results on a variety of datasets demonstrate that MahNN outperforms most of the state-of-the-arts for text classification.

CTS-LSTM: LSTM-based neural networks for correlatedtime series prediction

Correlated time series refer to multiple time series which are recorded simultaneously to monitor the changing of multiple observations in a whole system. Correlated time series prediction plays a significant role in many real-world applications to help people make reasonable decisions. Yet it is very challenging, because different from single time series, correlated time series show both intra-sequence temporal dependencies and inter-sequence spatial dependencies. In addition, correlated time series are also affected by external factors in actual scenarios. Although RNNs have been proved to be effective on sequential data modeling, existing related works only focus on sequential patterns in a single time series, failing to comprehensively consider the inter-dependencies among multiple time series, which is essential for correlated time series prediction. In this paper, we propose a novel variant of LSTM, named CTS-LSTM, to collectively forecast correlated time series. Specifically, spatial and temporal correlations are explicitly modeled and respectively maintained in cells to capture the complex non-linear patterns in correlated time series. A general interface for handling external factors is further designed to enhance forecasting performance of the model. Experiments are conducted on two types of real-world datasets, viz., civil aviation passenger demand data and air quality data. And our CTS-LSTM achieves at least 9.0%, 16.5% and 21.3% lower RMSE, MAE and MAPE compared to the state-of-the-art baselines.

Multilevel Information Fusion for Induction Motor Fault Diagnosis

Condition monitoring and fault diagnosis are of significance to improve the safety and reliability of motors, given their widespread applications in virtually every branch of the industry. Sequential data modeling based on recurrent neural network and its variants have drawn increasing attention because the temporal nature of motor signals can be well leveraged for motor analysis. One common drawback of prior research is that signals measured on motors are typically analyzed with a fixed time window, making it difficult to tradeoff between global state estimation and local feature extraction. This paper presents a deep-learning-based model termed multiresolution & multisensor fusion network for motor fault diagnosis, through multiscale analysis of motor vibration and stator current signals. Specifically, vibration and current signals are first segmented by analysis windows of varying lengths to create a new data stream for the joint representation and temporal encoding of the original sensor signals, based on two network structures: convolutional neural network and long short-term memory. The advantage of the developed method is that it automatically learns the discriminative features through the network training process, without requiring manual feature selection as is typically the case in prior methods. By considering the temporal dependence of the signals being analyzed, the developed multiresolution fusion technique not only improves the effectiveness of feature extraction but is also adaptive to varying motor speed. Two case studies demonstrate the advantages of the developed method.

Sequence to sequence learning with attention mechanism for short-term passenger flow prediction in large-scale metro system

The accurate short-term passenger flow prediction is of great significance for real-time public transit management, timely emergency response as well as systematical medium and long-term planning. In this paper, we propose an end-to-end deep learning framework that can simultaneously make multi-step predictions for all stations in a large scale metro system. A sequence to sequence model embedded with the attention mechanism forms the backbone of this framework. The sequence to sequence model consists of an encoder network and a decoder network, making it good at modeling sequential data with varying lengths and the attention mechanism further enhances its ability to capture long-range dependencies. We use the proposed framework to predict the number of passengers alighting at each station in the near future, given the number of passengers boarding at each station in the last few short-term periods. The large quantities of real-world data collected from Singapore’s metro system are used to validate the proposed model. In addition, a set of comparisons made among our model and other classical approaches evidently indicates that the proposed model is more scalable and robust than other baselines in making multi-step and network-wide predictions for short-term passenger flow.

Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition

Recurrent Neural Networks (RNNs) and their variants, such as Long-Short Term Memory (LSTM) networks, and Gated Recurrent Unit (GRU) networks, have achieved promising performance in sequential data modeling. The hidden layers in RNNs can be regarded as the memory units, which are helpful in storing information in sequential contexts. However, when dealing with high dimensional input data, such as video and text, the input-to-hidden linear transformation in RNNs brings high memory usage and huge computational cost. This makes the training of RNNs very difficult. To address this challenge, we propose a novel compact LSTM model, named as TR-LSTM, by utilizing the low-rank tensor ring decomposition (TRD) to reformulate the input-to-hidden transformation. Compared with other tensor decomposition methods, TR-LSTM is more stable. In addition, TR-LSTM can complete an end-to-end training and also provide a fundamental building block for RNNs in handling large input data. Experiments on real-world action recognition datasets have demonstrated the promising performance of the proposed TR-LSTM compared with the tensor-train LSTM and other state-of-the-art competitors.

Temporal Relational Ranking for Stock Prediction

Stock prediction aims to predict the future trends of a stock in order to help investors make good investment decisions. Traditional solutions for stock prediction are based on time-series models. With the recent success of deep neural networks in modeling sequential data, deep learning has become a promising choice for stock prediction.However, most existing deep learning solutions are not optimized toward the target of investment, i.e., selecting the best stock with the highest expected revenue. Specifically, they typically formulate stock prediction as a classification (to predict stock trends) or a regression problem (to predict stock prices). More importantly, they largely treat the stocks as independent of each other. The valuable signal in the rich relations between stocks (or companies), such as two stocks are in the same sector and two companies have a supplier-customer relation, is not considered.In this work, we contribute a new deep learning solution, namedRelational Stock Ranking(RSR), for stock prediction. Our RSR method advances existing solutions in two major aspects: (1) tailoring the deep learning models for stock ranking, and (2) capturing the stock relations in a time-sensitive manner. The key novelty of our work is the proposal of a new component in neural network modeling, namedTemporal Graph Convolution, which jointly models the temporal evolution and relation network of stocks. To validate our method, we perform back-testing on the historical data of two stock markets, NYSE and NASDAQ. Extensive experiments demonstrate the superiority of our RSR method. It outperforms state-of-the-art stock prediction solutions achieving an average return ratio of 98% and 71% on NYSE and NASDAQ, respectively.

Positive Sequential Data Modeling Using Continuous Hidden Markov Models Based on Inverted Dirichlet Mixtures

The hidden Markov model (HMM) has long been one of the most commonly used probability graph models for modeling sequential or time series data. It has been widely used in many fields ranging from speech recognition, face recognition, anomaly detection, to gene function prediction. In this paper, we theoretically propose a variant of the continuous HMM for modeling positive sequential data which are naturally generated in many real-life applications. In contrast with conventional HMMs which often use Gaussian distributions or Gaussian mixture models as the emission probability density, we adopt the inverted Dirichlet mixture model as the emission density to build the HMM. The consideration of inverted Dirichlet mixture model in our case is motivated by its superior modeling capability over Gaussian mixture models for modeling positive data according to several recent studies. In addition, we develop a convergence-guaranteed approach to learning the proposed inverted Dirichlet-based HMM through variational Bayes inference. The effectiveness of the proposed HMM is validated through both synthetic data sets and a real-world application regarding anomaly network intrusion detection. Based on the experimental results, the proposed inverted Dirichlet-based HMM is able to achieve the detection accuracy rates that are about 4%~9% higher than those ones obtained by the compared approaches.

Phonocardiographic Sensing Using Deep Learning for Abnormal Heartbeat Detection

Deep learning-based cardiac auscultation is of significant interest to the healthcare community as it can help reducing the burden of manual auscultation with automated detection of abnormal heartbeats. However, the problem of automatic cardiac auscultation is complicated due to the requirement of reliable and highly accurate systems, which are robust to the background noise in the heartbeat sound. In this paper, we propose a Recurrent Neural Networks (RNNs)-based automated cardiac auscultation solution. Our choice of RNNs is motivated by their great success of modeling sequential or temporal data even in the presence of noise. We explore the use of various RNN models, and demonstrate that these models significantly outperform the best reported results in the literature. We also present the run-time complexity of various RNNs, which provides insight about their complexity versus performance trade-offs.

Multi-kernel Gaussian process latent variable regression model for high-dimensional sequential data modeling

Modeling sequential data has been a hot research field for decades. One of the most challenge problems in this field is modeling real-world high-dimensional sequential data with limited training samples. This is mainly due to the following two reasons. First, if the dimension of the data is significantly greater then the number of the data, it may result in the over-fitting problem. Second, the dynamic behavior of the real-world data is very complex and difficult to approximate. To overcome these two problems, we propose a multi-kernel Gaussian process latent variable regression model for high-dimensional sequential data modeling and prediction. In our model, we design a regression model based on the Gaussian process latent variable model. Furthermore, a multi-kernel learning model is designed to automatically construct suitable nonlinear kernel for various types of sequential data. We evaluate the effectiveness of our method using two types of real-world high-dimensional sequential data, including the human motion data and the motion texture video data. In addition, our method is compared with several representative sequential data modeling methods. Experimental results show that our method achieves promising modeling capability and is capable of predict human motion and texture video with higher quality.

DeepGesture: Deep learning-based gesture recognition scheme using motion sensors

Recent advancement in smart phones and sensor technology has promoted research in gesture recognition. This has made designing of efficient gesture interface easy. However, human activity recognition (HAR) through gestures is not trivial since each person may pose the same gesture differently. In this paper, we propose deepGesture algorithm, a new arm gesture recognition method based on gyroscope and accelerometer sensors using deep convolution and recurrent neural networks. This method uses four deep convolution layers to automate feature learning in raw sensor data. The features of the convolution layers are used as input of the gated recurrent unit (GRU) which is based on the state-of-the-art recurrent neural network (RNN) structure to capture long-term dependency and model sequential data. The input data of the proposed algorithm is obtained through motion sequence data extracted using a wrist-type smart band device equipped with gyroscope and accelerometer sensors. The data is initially segmented in fixed length segments. The segmented data is labeled and we construct the database. Then the labeled data is used in our learning algorithm. To verify the applicability of the algorithm, several experiments have been performed to measure the accuracy of gesture classification. Compared to the human activity recognition method, our experimental results show that the proposed deepGesture algorithm can increase the average F1-score for recognition of nine defined arm gestures by 6%.

STSCR: Exploring spatial-temporal sequential influence and social information for location recommendation

Abstract One remarkable progress in spatial data analysis field lies on location-based social networks (LBSNs), where location recommendation has become an important way to help people locate the interesting places. When providing the location recommendations for users, spatial-temporal sequence of locations on users check-in behaviors have been intensively studied based on the fact that the human movement usually exhibits a pattern of spatial-temporal sequence. However, despite that some works have been proposed to focus on the spatial-temporal correlation among the sequences of users checked-in locations, they still suffer to the following their limitations: (1) these works are unable to handle time intervals between nearby check-in behaviors properly in modeling sequential data. In fact, POIs also have different probabilities of being checked in different time periods; (2) these works fail to employ the social relations adequately to improve final recommendation results especially when the check-in data is sparse; (3) these works are essentially based on the point-wise theory that aims to regress a real-valued score, whereas few work attempt to build a ranking-based estimator for recommendation. To solve the problems mentioned above, a new location recommendation model called Spatial-Temporal aware Social Collaborative Ranking(STSCR) model is proposed to explore the impact of time, spatial-temporal sequential influence and social influence. In particular, the proposed model is built upon a unified tensor factorization framework in which the ne-grained modeling of user-location, user-friend, friend-location, location-time, and location-location interactions. Then, the Bayesian personalized ranking technique is used to optimized the loss function of unified tensor factorization and fit the partial order of locations. As is shown in the experimental results derived from real datasets, our proposed STSCR achieves better recommendation performances than other state-of-the-art location recommendation algorithms.

Recurrent Neural Network for Classification of Snoring and Non-Snoring Sound Events.

Obstructive sleep apnea (OSA) is a disorder that affects up to 38% of the western population. It is characterized by repetitive episodes of partial or complete collapse of the upper airway during sleep. These episodes are almost always accompanied by loud snoring. Questionnaires such as STOP-BANG exploit snoring to screen for OSA. However, they are not quantitative and thus do not exploit its full potential. A method for automatic detection of snoring in whole-night recordings is required to enable its quantitative evaluation. In this study, we propose such a method. The centerpiece of the proposed method is a recurrent neural network for modeling of sequential data with variable length. Mel-frequency cepstral coefficients, which were extracted from snoring and non-snoring sound events, were used as inputs to the proposed network. A total of 20 subjects referred to clinical sleep recording were also recorded by a microphone that was placed 70 cm from the top end of the bed. These recordings were used to assess the performance of the proposed method. When it comes to the detection of snoring events, our results show that the proposed method has an accuracy of 95%, sensitivity of 92%, and specificity of 98%. In conclusion, our results suggest that the proposed method may improve the process of snoring detection and with that the process of OSA screening. Follow-up clinical studies are required to confirm this potential.

Latent subspace modeling of sequential data under the maximum entropy discrimination framework

Abstract Hidden Markov models (HMMs) are a popular approach for modeling continuous sequential data, typically based on the assumption of Gaussian-distributed observations. A significant issue HMMs with Gaussian conditional densities are confronted with concerns effectively modeling high-dimensional observations, without getting prone to overfitting or singularities. To this end, one can resort to extracting lower-dimensional latent variable representations of the observed high-dimensional data, as part of the inference algorithm of the postulated HMM. Factor analysis (FA) is a well-established linear latent variable scheme that can be employed for this purpose; its functionality consists in modeling the covariances between the elements of multivariate observations under a set of linear assumptions. Recently, it has been proposed that FA can be effectively generalized under an efficient large-margin Bayesian inference perspective, namely maximum entropy discrimination (MED). This work capitalizes on these recent findings to derive an effective HMM-driven sequential data modeling framework for high-dimensional data. Our proposed approach extracts lower-dimensional latent variable representations of observed high-dimensional data, taking into account the large-margin principle. On this basis, it postulates that the data temporal dynamics are conditional to the inferred values of these latent variables. We devise efficient mean-field inference algorithms for our model, and exhibit its advantages through a set of experiments.

Sentic LSTM: a Hybrid Network for Targeted Aspect-Based Sentiment Analysis

Sentiment analysis has emerged as one of the most popular natural language processing (NLP) tasks in recent years. A classic setting of the task mainly involves classifying the overall sentiment polarity of the inputs. However, it is based on the assumption that the sentiment expressed in a sentence is unified and consistent, which does not hold in the reality. As a fine-grained alternative of the task, analyzing the sentiment towards a specific target and aspect has drawn much attention from the community for its more practical assumption that sentiment is dependent on a particular set of aspects and entities. Recently, deep neural models have achieved great successes on sentiment analysis. As a functional simulation of the behavior of human brains and one of the most successful deep neural models for sequential data, long short-term memory (LSTM) networks are excellent in learning implicit knowledge from data. However, it is impossible for LSTM to acquire explicit knowledge such as commonsense facts from the training data for accomplishing their specific tasks. On the other hand, emerging knowledge bases have brought a variety of knowledge resources to our attention, and it has been acknowledged that incorporating the background knowledge is an important add-on for many NLP tasks. In this paper, we propose a knowledge-rich solution to targeted aspect-based sentiment analysis with a specific focus on leveraging commonsense knowledge in the deep neural sequential model. To explicitly model the inference of the dependent sentiment, we augment the LSTM with a stacked attention mechanism consisting of attention models for the target level and sentence level, respectively. In order to explicitly integrate the explicit knowledge with implicit knowledge, we propose an extension of LSTM, termed Sentic LSTM. The extended LSTM cell includes a separate output gate that interpolates the token-level memory and the concept-level input. In addition, we propose an extension of Sentic LSTM by creating a hybrid of the LSTM and a recurrent additive network that simulates sentic patterns. In this paper, we are mainly concerned with a joint task combining the target-dependent aspect detection and targeted aspect-based polarity classification. The performance of proposed methods on this joint task is evaluated on two benchmark datasets. The experiment shows that the combination of proposed attention architecture and knowledge-embedded LSTM could outperform state-of-the-art methods in two targeted aspect sentiment tasks. We present a knowledge-rich solution for the task of targeted aspect-based sentiment analysis. Our model can effectively incorporate the commonsense knowledge into the deep neural network and be trained in an end-to-end manner. We show that the two-step attentive neural architecture as well as the proposed Sentic LSTM and H-Sentic-LSTM can achieve an improved performance on resolving the aspect categories and sentiment polarity for a targeted entity in its context over state-of-the-art systems.

Dynamic Hybrid Random Fields for the Probabilistic Graphical Modeling of Sequential Data: Definitions, Algorithms, and an Application to Bioinformatics

The paper introduces a dynamic extension of the hybrid random field (HRF), called dynamic HRF (D-HRF). The D-HRF is aimed at the probabilistic graphical modeling of arbitrary-length sequences of sets of (time-dependent) discrete random variables under Markov assumptions. Suitable maximum likelihood algorithms for learning the parameters and the structure of the D-HRF are presented. The D-HRF inherits the computational efficiency and the modeling capabilities of HRFs, subsuming both dynamic Bayesian networks and Markov random fields. The behavior of the D-HRF is first evaluated empirically on synthetic data drawn from probabilistic distributions having known form. Then, D-HRFs (combined with a recurrent autoencoder) are successfully applied to the prediction of the disulfide-bonding state of cysteines from the primary structure of proteins in the Protein Data Bank.

Symbolic pattern recognition for sequential data

ABSTRACTSources of sequential data surround and pervade our lives. Our bodies continuously generate sequential data such as heart rate and blood pressure. In global finance, stock indices and currency exchange rates change every second. The movement of clouds, the coordinates of the planets, the score of a soccer game, etc., are all examples of sequential data transitioning from one state to another in regular time steps. There is a mature body of literature related to modeling time-dependent sequential data, or time series, in which every model relies upon specific assumptions for applicability to data. However, there other kinds of sequential data that are not collected with respect to time; for example, DNA sequences (if we ignore the gene mutations). Modeling of this type of sequential data does not have a body of literature that is as mature. This is somewhat perplexing, since sequential data modeling should cover both time-dependent and non-time-dependent data.In this study, we introduce a framework called Symbolic Pattern Recognition for modeling pattern transition behavior of sequential data that is expressed by a finite alphabet of symbols. Our framework can be used to characterize, predict, simulate, cluster, and classify multiple series based on their observed pattern transition behaviors.We document our proposed framework and apply it to perform unsupervised clustering of 13 different species of mollusks based on their DNA sequences. Our model correctly clusters the mollusks into their respective genera, matching results obtained via more traditional, supervised genetic analysis.

Lithium-ion Battery Remaining Useful Life Prediction with Long Short-term Memory Recurrent Neural Network

Lithium-ion batteries play critical roles in many electronic devices. It is necessary to develop a reliable and accurate remaining useful life (RUL) prediction approach to provide timely maintenance or replacement of battery systems. A novel RUL prediction approach based on Long Short-term Memory (LSTM) Recurrent Neural Network (RNN) is proposed in this paper. LSTM is able to capture long-term dependencies and model sequential data among the capacity degradation of lithium-ion batteries. The advantages of our proposed method include: 1) obtaining high prediction accuracy without accurate physics-based model or expertise and 2) decreasing the cumulation errors by multi-step ahead prediction each time, while traditional RUL method predicts one-step ahead once and then uses the current estimated value to predict next one, which causes cumulation errors increased. The Center for Advanced Life Cycle Engineering (CALCE) battery datasets are used to demonstrate the effectiveness of the proposed method. The results show that, compared with echo state networks (ESN), the proposed method has higher accuracy, more stable and reliable performance for lithium-ion batteries RUL prediction.