Modeling Sequential Data Research Articles

Unsupervised modeling and feature selection of sequential spherical data through nonparametric hidden Markov models

Unsupervised modeling and feature selection of sequential spherical data through nonparametric hidden Markov models

Continual Sequence Modeling With Predictive Coding.

Recurrent neural networks (RNNs) have been proved very successful at modeling sequential data such as language or motions. However, these successes rely on the use of the backpropagation through time (BPTT) algorithm, batch training, and the hypothesis that all the training data are available at the same time. In contrast, the field of developmental robotics aims at uncovering lifelong learning mechanisms that could allow embodied machines to learn and stabilize knowledge in continuously evolving environments. In this article, we investigate different RNN designs and learning methods, that we evaluate in a continual learning setting. The generative modeling task consists in learning to generate 20 continuous trajectories that are presented sequentially to the learning algorithms. Each method is evaluated according to the average prediction error over the 20 trajectories obtained after complete training. This study focuses on learning algorithms with low memory requirements, that do not need to store past information to update their parameters. Our experiments identify two approaches especially fit for this task: conceptors and predictive coding. We suggest combining these two mechanisms into a new proposed model that we label PC-Conceptors that outperforms the other methods presented in this study.

Conditional Random Fields for Multiview Sequential Data Modeling.

Recently, multiview learning has been increasingly focused on machine learning. However, most existing multiview learning methods cannot directly deal with multiview sequential data, in which the inherent dynamical structure is often ignored. Especially, most traditional multiview machine learning methods assume that the items at different time slices within a sequence are independent of each other. In order to solve this problem, we propose a new multiview discriminant model based on conditional random fields (CRFs) to model multiview sequential data, called multiview CRF. It inherits the advantages of CRFs that build a relationship between items in each sequence. Moreover, by introducing specific features designed on the CRFs for multiview data, the multiview CRF not only considers the relationship among different views but also captures the correlation between the features from the same view. Particularly, some features can be reused or divided into different views to build an appropriate size of feature space. This helps to avoid underfitting problems caused by too small feature space or overfitting problems caused by too large feature space. In order to handle large-scale data, we use the stochastic gradient method to speed up our model. The experimental results on the text and video data illustrate the superiority of the proposed model.

A recurrent neural network architecture to model physical activity energy expenditure in older people

Through the quantification of physical activity energy expenditure (PAEE), health care monitoring has the potential to stimulate vital and healthy ageing, inducing behavioural changes in older people and linking these to personal health gains. To be able to measure PAEE in a health care perspective, methods from wearable accelerometers have been developed, however, mainly targeted towards younger people. Since elderly subjects differ in energy requirements and range of physical activities, the current models may not be suitable for estimating PAEE among the elderly. Furthermore, currently available methods seem to be either simple but non-generalizable or require elaborate (manual) feature construction steps. Because past activities influence present PAEE, we propose a modeling approach known for its ability to model sequential data, the recurrent neural network (RNN). To train the RNN for an elderly population, we used the growing old together validation (GOTOV) dataset with 34 healthy participants of 60 years and older (mean 65 years old), performing 16 different activities. We used accelerometers placed on wrist and ankle, and measurements of energy counts by means of indirect calorimetry. After optimization, we propose an architecture consisting of an RNN with 3 GRU layers and a feedforward network combining both accelerometer and participant-level data. Our efforts included switching mean to standard deviation for down-sampling the input data and combining temporal and static data (person-specific details such as age, weight, BMI). The resulting architecture produces accurate PAEE estimations while decreasing training input and time by a factor of 10. Subsequently, compared to the state-of-the-art, it is capable to integrate longer activity data which lead to more accurate estimations of low intensity activities EE. It can thus be employed to investigate associations of PAEE with vitality parameters of older people related to metabolic and cognitive health and mental well-being.

Event detection in temporal social networks using a higher-order network model.

Event detection is one of the most important areas of complex network research. It aims to identify abnormal points in time corresponding to social events. Traditional methods of event detection, based on first-order network models, are poor at describing the multivariate sequential interactions of components in complex systems and at accurately identifying anomalies in temporal social networks. In this article, we propose two valid approaches, based on a higher-order network model, namely, the recovery higher-order network algorithm and the innovation higher-order network algorithm, to help with event detection in temporal social networks. Given binary sequential data, we take advantage of chronological order to recover the multivariate sequential data first. Meanwhile, we develop new multivariate sequential data using logical sequence. Through the efficient modeling of multivariate sequential data using a higher-order network model, some common multivariate interaction patterns are obtained, which are used to determine the anomaly degree of a social event. Experiments in temporal social networks demonstrate the significant performance of our methods finally. We believe that our methods could provide a new perspective on the interplay between event detection and the application of higher-order network models to temporal networks.

Differential impact of the COVID-19 pandemic on laboratory reporting of norovirus and Campylobacter in England: A modelling approach.

The COVID-19 pandemic has impacted surveillance activities for multiple pathogens. Since March 2020, there was a decline in the number of reports of norovirus and Campylobacter recorded by England's national laboratory surveillance system. The aim is to estimate and compare the impact of the COVID-19 pandemic on norovirus and Campylobacter surveillance data in England. We utilised two quasi-experimental approaches based on a generalised linear model for sequential count data. The first approach estimates overall impact and the second approach focuses on the impact of specific elements of the pandemic response (COVID-19 diagnostic testing and control measures). The following time series (27, 2015-43, 2020) were used: weekly laboratory-confirmed norovirus and Campylobacter reports, air temperature, conducted Sars-CoV-2 tests and Index of COVID-19 control measures stringency. The period of Sars-CoV-2 emergence and subsequent sustained transmission was associated with persistent reductions in norovirus laboratory reports (p = 0.001), whereas the reductions were more pronounced during pandemic emergence and later recovered for Campylobacter (p = 0.075). The total estimated reduction was 47% - 79% for norovirus (12-43, 2020). The total reduction varied by time for Campylobacter, e.g. 19% - 33% in April, 1% - 7% in August. Laboratory reporting of norovirus was more adversely impacted than Campylobacter by the COVID-19 pandemic. This may be partially explained by a comparatively stronger effect of behavioural interventions on norovirus transmission and a relatively greater reduction in norovirus testing capacity. Our study underlines the differential impact a pandemic may have on surveillance of gastrointestinal infectious diseases.

Simultaneous positive sequential vectors modeling and unsupervised feature selection via continuous hidden Markov models

Since positive data vectors are often naturally generated in various real-life applications, positive vectors modeling has become an important research topic. In this article, we tackle the problem of modeling positive sequential vectors through continuous hidden Markov models (HMMs). Motivated by several recent studies in which the generalized inverted Dirichlet (GID) distribution has provided better performance than the Gaussian distribution for modeling positive data, instead of adopting Gaussian mixture models (GMM) as the emission density for conventional continuous HMMs, we theoretically propose a novel HMM by considering the mixture of GID distributions as the emission density. Moreover, to cope with high-dimensional data which may contain irrelevant features, an unsupervised localized feature selection method is incorporated with our model, which results in a unified framework that can simultaneously perform positive sequential data modeling and feature selection. To learn the proposed model, we develop a convergence-guaranteed algorithm based on variational Bayes. The advantages of our model are demonstrated through both simulated data sets and a real-life application about human action recognition.

Application of a long short-term memory neural network for modeling transonic buffet aerodynamics

In the present work, a reduced-order modeling (ROM) framework based on a long short-term memory (LSTM) neural network is applied for the prediction of transonic buffet aerodynamics. This type of network has a high potential for modeling sequential data, which is favorable for capturing the time-delayed effects associated with unsteady aerodynamics. Therefore, the nonlinear identification procedure as well as the generalization of the resulting ROM are presented. Further, a Monte-Carlo-based training procedure is performed in order to estimate statistical errors. The training data set for the ROM is provided by means of forced-motion unsteady Reynolds-averaged Navier Stokes (URANS) simulation. Subsequent to the training process, the ROM is applied for the computation of time-varying integral quantities such as aerodynamic force and moment coefficients. The most challenging aspect when considering buffet aerodynamics is given by the reproduction of the self-sustained unsteadiness of the buffeting flow. Even without any external excitation, the flow is characterized by large shock-boundary layer interaction, resulting in shock movement and flow separation. Finally, the performance of the trained network is demonstrated by predicting the aerodynamic loads of the NACA0012 airfoil considered at transonic freestream conditions. Therefore, the airfoil is excited by a forced pitching motion beyond the buffet-critical angle of attack. A comparison with a full-order computational fluid dynamics (CFD) solution shows that the essential characteristics of the nonlinear buffet phenomenon are captured by the ROM method.

Recurrent neural networks with long term temporal dependencies in machine tool wear diagnosis and prognosis

Data-driven approaches for machine tool wear diagnosis and prognosis are gaining attention in the past few years. The goal of our study is to advance the adaptability, flexibility, prediction performance, and prediction horizon for online monitoring and prediction. This paper proposes the use of a recent deep learning method, based on Gated Recurrent Neural Network architecture, including Long Short Term Memory (LSTM), which try to captures long-term dependencies than regular Recurrent Neural Network method for modeling sequential data, and also the mechanism to realize the online diagnosis and prognosis and remaining useful life (RUL) prediction with indirect measurement collected during the manufacturing process. Existing models are usually tool-specific and can hardly be generalized to other scenarios such as for different tools or operating environments. Different from current methods, the proposed model requires no prior knowledge about the system and thus can be generalized to different scenarios and machine tools. With inherent memory units, the proposed model can also capture long-term dependencies while learning from sequential data such as those collected by condition monitoring sensors, which means it can be accommodated to machine tools with varying life and increase the prediction performance. To prove the validity of the proposed approach, we conducted multiple experiments on a milling machine cutting tool and applied the model for online diagnosis and RUL prediction. Without loss of generality, we incorporate a system transition function and system observation function into the neural net and trained it with signal data from a minimally intrusive vibration sensor. The experiment results showed that our LSTM-based model achieved the best overall accuracy among other methods, with a minimal Mean Square Error (MSE) for tool wear prediction and RUL prediction respectively.

Study on a CNN-HMM Approach for Audio-Based Musical Chord Recognition

Convolutional Neural Network (CNN) has shown its strength in image processing task, and Hidden Markov model (HMM) is a powerful tool for modeling sequential data. This paper presents a new architecture for audio-based chord recognition using a CNN-HMM mixture model. This architecture replaces the Gaussian mixture model (GMM) and Deep Neural Network (DNN) layers of GMM-HMM and DNN-HMM models with CNN. The model performance is evaluated through a dataset using different combinations of chroma vectors (STFT, CQT, CENS) as features, based on that, a scale recognition sub-model is tested.

Slag foaming estimation in the electric arc furnace using machine learning based long short-term memory networks

Slag foaming is a key factor in terms of quality and productivity in the electric arc furnace (EAF) steelmaking process. Optimal control of slag foaming is required, but is difficult due to the absence of practical on-line measuring methods and the broad process variability. In this study, a soft sensor model, which correlates the influential process variables with the slag foaming height, was developed by using machine learning based long short-term memory (LSTM) networks for modeling sequential and nonlinear data. The developed model was validated using actual steelmaking dataset in terms of performance metrics such as the root mean square error (RMSE), coefficient of determination (R2), and correlation coefficient corresponding to a value for a SS400 carbon steel grade to be 42.3, 0.905, and 0.963, respectively. In order to evaluate the general applicability of the developed model for other steel grades, data for A615 and S355 steel grades were also applied and found to satisfy the benchmark standards indicating that the developed model can be applied to the broad range of other steel grades. Sensitivity-based Pruning (SBP) on the model shows that electricity, carbon and oxygen are the most influential process variables to the slag foaming height and could potentially be used to promote enhanced optimization in terms of energy saving and cost-efficiency for the EAF steelmaking process.

KDnet-RUL: A Knowledge Distillation Framework to Compress Deep Neural Networks for Machine Remaining Useful Life Prediction

Machine remaining useful life (RUL) prediction is vital in improving the reliability of industrial systems and reducing maintenance cost. Recently, long short-term memory (LSTM) based algorithms have achieved state-of-the-art performance for RUL prediction due to their strong capability of modeling sequential sensory data. In many cases, the RUL prediction algorithms are required to be deployed on edge devices to support real-time decision making, reduce the data communication cost, and preserve the data privacy. However, the powerful LSTM-based methods which have high complexity cannot be deployed to edge devices with limited computational power and memory. To solve this problem, we propose a knowledge distillation framework, entitled KDnet-RUL, to compress a complex LSTM-based method for RUL prediction. Specifically, it includes a generative adversarial network based knowledge distillation (GAN-KD) for disparate architecture knowledge transfer, a learning-during-teaching based knowledge distillation (LDT-KD) for identical architecture knowledge transfer, and a sequential distillation upon LDT-KD for complicated datasets. We leverage simple and complicated datasets to verify the effectiveness of the proposed KDnet-RUL. The results demonstrate that the proposed method significantly outperforms state-of-the-art KD methods. The compressed model with 12.8 times less weights and 46.2 times less total float point operations even achieves a comparable performance with the complex LSTM model for RUL prediction.

Spatiotemporal Co-Attention Recurrent Neural Networks for Human-Skeleton Motion Prediction.

Human motion prediction aims to generate future motions based on the observed human motions. Witnessing the success of Recurrent Neural Networks (RNN) in modeling sequential data, recent works utilize RNNs to model human-skeleton motions on the observed motion sequence and predict future human motions. However, these methods disregard the existence of the spatial coherence among joints and the temporal evolution among skeletons, which reflects the crucial characteristics of human motions in spatiotemporal space. To this end, we propose a novel Skeleton-Joint Co-Attention Recurrent Neural Networks (SC-RNN) to capture the spatial coherence among joints, and the temporal evolution among skeletons simultaneously on a skeleton-joint co-attention feature map in spatiotemporal space. First, a skeleton-joint feature map is constructed as the representation of the observed motion sequence. Second, we design a new Skeleton-Joint Co-Attention (SCA) mechanism to dynamically learn a skeleton-joint co-attention feature map of this skeleton-joint feature map, which can refine the useful observed motion information to predict one future motion. Third, a variant of GRU embedded with SCA collaboratively models the human-skeleton motion and human-joint motion in spatiotemporal space by regarding the skeleton-joint co-attention feature map as the motion context. Experimental results of human motion prediction demonstrate that the proposed method outperforms the competing methods.

Text classification by untrained sentence embeddings

Recurrent Neural Networks (RNNs) represent a natural paradigm for modeling sequential data like text written in natural language. In fact, RNNs and their variations have long been the architecture of choice in many applications, however in practice they require the use of labored architectures (such as gating mechanisms) and computationally heavy training processes. In this paper we address the question of whether it is possible to generate sentence embeddings via completely untrained recurrent dynamics, on top of which to apply a simple learning algorithm for text classification. This would allow to obtain extremely efficient models in terms of training time. Our work investigates the extent to which this approach can be used, by analyzing the results on different tasks. Finally, we show that, within certain limits, it is possible to build extremely efficient models for text classification that remain competitive in accuracy with reference models in the state-of-the-art.

Analyzing the Short-Term Dependency in Ultra-High Magnetic Response Systems - Modeling Sequential Data with Non-Recurrent Neural Networks

Recurrent neural network (RNN) is a popular modeling choice for sequential data. However, empirical experience shows RNNs are often difficult and time-consuming to tune and customize. It drives practitioners to replace RNNs with non-recurrent neural networks which presented comparable performances in some cases. The success of using non-recurrent neural networks to model sequential data indicates the short-term dependency among sequential data. In this paper, we systematically analyze the short-term dependency in ultra-high magnetic response systems (UHMR) with partial system knowledge based on the observability criterion of the dynamic systems. Moreover, we show that the sequential data in the UHMR system only have 2-step dependency. This result indicates that any consecutive three steps in an experiment form a datum to train a feed-forward neural network (FFNN). Therefore, sufficient data can be collected within a small number of experiments. Based on the analysis, we train a feed-forward neural network to model the UHMR system based on the sequential data from four experiments. Through proper data pre-processing, the FFNN model can predict the system performance with bounded mean absolute error.

Implementation of Deep Learning Predictor (LSTM) Algorithm for Human Mobility Prediction

The studies of human mobility prediction in mobile computing area gained due to the availability of large-scale dataset contained history of location trajectory. Previous work has been proposed many solutions for increasing of human mobility prediction result accuration, however, only few researchers have addressed the issue of<em> </em>human mobility for implementation of LSTM networks. This study attempted to use classical methodologies by combining LSTM and DBSCAN because those algorithms can tackle problem in human mobility, including large-scale sequential data modeling and number of clusters of arbitrary trajectory identification. The method of research consists of DBSCAN for clustering, long short-term memory (LSTM) algorithm for modelling and prediction, and Root Mean Square Error (RMSE) for evaluation. As the result,<em> </em>the prediction error or RMSE value reached score 3.551 by setting LSTM with parameter of <em>epoch</em> and <em>batch_size</em> is 100 and 20 respectively.

Occurrence prediction of cotton pests and diseases by bidirectional long short-term memory networks with climate and atmosphere circulation

The occurrence of crop pests and diseases always affects the development of agriculture seriously, while pest meteorology showed that climate is important in affecting the occurrence. Recently, recurrent neural network (RNN) has been broadly applied in various fields, which was designed for modeling sequential data and has been testified to be quite efficient in time series problem. This paper proposes to use bi-directional RNN with long short-term memory (LSTM) units for predicting the occurrence of cotton pests and diseases with climate factors. First, the problem of occurrence prediction of pests and diseases is formulated as time series prediction. Then the bi-directional LSTM network (Bi-LSTM) is adopted to solve the problem, which can capture long-term dependencies on the past and future contexts of sequential data. Experimental results showed that Bi-LSTM shows good performance on the occurrence prediction of pests and diseases in cotton fields, and yields an Area Under the Curve (AUC) of 0.95. This work further verified that climate indeed have strong impact on the occurrence of pests and diseases, and circulation parameters also have certain influence.

Modeling Event Propagation via Graph Biased Temporal Point Process.

Temporal point process is widely used for sequential data modeling. In this article, we focus on the problem of modeling sequential event propagation in graph, such as retweeting by social network users and news transmitting between websites. Given a collection of event propagation sequences, the conventional point process model considers only the event history, i.e., embed event history into a vector, not the latent graph structure. We propose a graph biased temporal point process (GBTPP) leveraging the structural information from graph representation learning, where the direct influence between nodes and indirect influence from event history is modeled. Moreover, the learned node embedding vector is also integrated into the embedded event history as side information. Experiments on a synthetic data set and two real-world data sets show the efficacy of our model compared with conventional methods and state-of-the-art ones.

An incrementally cascaded broad learning framework to facial landmark tracking

Facial landmark tracking often adopts static models to generically fit per frame of video. This is considered inappropriate since such models ignore the informative correlation between previous and current frames. Moreover, most of these methods fail to balance the speed and accuracy for video-based facial landmark tracking simultaneously. In this paper, we propose an efficient online framework for video-based face alignment, named Incrementally Cascaded Broad Learning framework (ICBL). ICBL aims to continuously enhance the prediction capability of tracking model for sequential data. It is capable of learning the spatial appearance on specific-person statistics from continuous facial frames and using such knowledge to incrementally tune a cascade of regressors in parallel. To achieve this goal, we approximate the facial shape space by sampling from a dynamic distribution which is continuously updated by person-specific statistics from the tracked facial frames. This dramatically facilitates cascade regression to incrementally update all cascade-regressors in parallel, thus allowing a fast update of the whole model. Furthermore, we successfully incorporate both the linear and non-linear mappings into our parallel cascade framework and introduce Broad Learning (BL) algorithm as a solution for them simultaneously. Experimental results on the most popular and large-scale benchmark for facial landmark tracking show highly competitive performance of proposed ICBL in comparisons with the state-of-the-arts. The code of our ICBL framework has been available fromhttps://github.com/CaifengLiu/Facial-landmark-tracking-by-ICBL.

Span-Based Neural Buffer: Towards Efficient and Effective Utilization of Long-Distance Context for Neural Sequence Models

Neural sequence model, though widely used for modeling sequential data such as the language model, has sequential recency bias (Kuncoro et al. 2018) to the local context, limiting its full potential to capture long-distance context. To address this problem, this paper proposes augmenting sequence models with a span-based neural buffer that efficiently represents long-distance context, allowing a gate policy network to make interpolated predictions from both the neural buffer and the underlying sequence model. Training this policy network to utilize long-distance context is however challenging due to the simple sentence dominance problem (Marvin and Linzen 2018). To alleviate this problem, we propose a novel training algorithm that combines an annealed maximum likelihood estimation with an intrinsic reward-driven reinforcement learning. Sequence models with the proposed span-based neural buffer significantly improve the state-of-the-art perplexities on the benchmark Penn Treebank and WikiText-2 datasets to 43.9 and 35.2 respectively. We conduct extensive analysis and confirm that the proposed architecture and the training algorithm both contribute to the improvements.