Time Series Classification Tasks Research Articles

Theory and Algorithms for Shapelet-Based Multiple-Instance Learning.

We propose a new formulation of multiple-instance learning (MIL), in which a unit of data consists of a set of instances called a bag. The goal is to find a good classifier of bags based on the similarity with a "shapelet" (or pattern), where the similarity of a bag with a shapelet is the maximum similarity of instances in the bag. In previous work, some of the training instances have been chosen as shapelets with no theoretical justification. In our formulation, we use all possible, and thus infinitely many, shapelets, resulting in a richer class of classifiers. We show that the formulation is tractable, that is, it can be reduced through linear programming boosting (LPBoost) to difference of convex (DC) programs of finite (actually polynomial) size. Our theoretical result also gives justification to the heuristics of some previous work. The time complexity of the proposed algorithm highly depends on the size of the set of all instances in the training sample. To apply to the data containing a large number of instances, we also propose a heuristic option of the algorithm without the loss of the theoretical guarantee. Our empirical study demonstrates that our algorithm uniformly works for shapelet learning tasks on time-series classification and various MIL tasks with comparable accuracy to the existing methods. Moreover, we show that the proposed heuristics allow us to achieve the result in reasonable computational time.

GRATIS: GeneRAting TIme Series with diverse and controllable characteristics

AbstractThe explosion of time series data in recent years has brought a flourish of new time series analysis methods, for forecasting, clustering, classification and other tasks. The evaluation of these new methods requires either collecting or simulating a diverse set of time series benchmarking data to enable reliable comparisons against alternative approaches. We propose GeneRAting TIme Series with diverse and controllable characteristics, named GRATIS, with the use of mixture autoregressive (MAR) models. We simulate sets of time series using MAR models and investigate the diversity and coverage of the generated time series in a time series feature space. By tuning the parameters of the MAR models, GRATIS is also able to efficiently generate new time series with controllable features. In general, as a costless surrogate to the traditional data collection approach, GRATIS can be used as an evaluation tool for tasks such as time series forecasting and classification. We illustrate the usefulness of our time series generation process through a time series forecasting application.

FilterNet: A Many-to-Many Deep Learning Architecture for Time Series Classification.

In this paper, we present and benchmark FilterNet, a flexible deep learning architecture for time series classification tasks, such as activity recognition via multichannel sensor data. It adapts popular convolutional neural network (CNN) and long short-term memory (LSTM) motifs which have excelled in activity recognition benchmarks, implementing them in a many-to-many architecture to markedly improve frame-by-frame accuracy, event segmentation accuracy, model size, and computational efficiency. We propose several model variants, evaluate them alongside other published models using the Opportunity benchmark dataset, demonstrate the effect of model ensembling and of altering key parameters, and quantify the quality of the models’ segmentation of discrete events. We also offer recommendations for use and suggest potential model extensions. FilterNet advances the state of the art in all measured accuracy and speed metrics when applied to the benchmarked dataset, and it can be extensively customized for other applications.

Edge4TSC: Binary Distribution Tree-Enabled Time Series Classification in Edge Environment.

In the past decade, time series data have been generated from various fields at a rapid speed, which offers a huge opportunity for mining valuable knowledge. As a typical task of time series mining, Time Series Classification (TSC) has attracted lots of attention from both researchers and domain experts due to its broad applications ranging from human activity recognition to smart city governance. Specifically, there is an increasing requirement for performing classification tasks on diverse types of time series data in a timely manner without costly hand-crafting feature engineering. Therefore, in this paper, we propose a framework named Edge4TSC that allows time series to be processed in the edge environment, so that the classification results can be instantly returned to the end-users. Meanwhile, to get rid of the costly hand-crafting feature engineering process, deep learning techniques are applied for automatic feature extraction, which shows competitive or even superior performance compared to state-of-the-art TSC solutions. However, because time series presents complex patterns, even deep learning models are not capable of achieving satisfactory classification accuracy, which motivated us to explore new time series representation methods to help classifiers further improve the classification accuracy. In the proposed framework Edge4TSC, by building the binary distribution tree, a new time series representation method was designed for addressing the classification accuracy concern in TSC tasks. By conducting comprehensive experiments on six challenging time series datasets in the edge environment, the potential of the proposed framework for its generalization ability and classification accuracy improvement is firmly validated with a number of helpful insights.

Robust IoT time series classification with data compression and deep learning

Internet of Things (IoT) and wearable systems are very resource limited in terms of power, memory, bandwidth and processor performance. Sensor time series compression can be regarded as a direct way to use memory and bandwidth resources efficiently. On the other hand, the time series classification has recently attracted great attention and has found numerous potential uses in areas such as finance, industry and healthcare. This paper investigates the effect of lossy compression techniques on the time series classification task using deep neural networks. Furthermore, this paper proposes an efficient compression approach for univariate and multivariate time series that combines the lifting implementation of the discrete wavelet transform with an error-bound compressor, namely Squeeze (SZ), to attain an optimal trade-off between data compression and data quality.

Dynamic Multi-Scale Convolutional Neural Network for Time Series Classification

Time series classification is an essential task in many real-world application domains. As a popular deep learning network, convolutional neural networks have achieved excellent performance in time series classification tasks. The filters of the convolutional neural networks are fixed length and shared by each sample. However, each time series usually has different time scale features. Therefore, convolutional neural networks are not capable of extracting multi-scale features for each sample flexibly. In this paper, we propose dynamic multi-scale convolutional neural network to extract multi-scale feature representations existing in each time series dynamically. Specifically, we design a variable-length filters generator to produce a set of variable-length filters conditioned on the input time series. To make model differentiable, we use the learnable soft masks to control the lengths of variable-length filters. Therefore, the feature representations of different time scales can be captured through the variable-length filters. Then, the max-over-time pooling is used to select the most discriminative local patterns. Finally, the fully connected layer with softmax output is employed to calculate the final probability distribution for each class. Experiments conducted on extensive time series datasets show that our approach can improve the performance of time series classification through the learning of variable-length filters. Furthermore, we demonstrate the effectiveness of dynamically learning variable-length filters for each sample through the visualization analysis.

Quantitative and Qualitative Analysis of Time-Series Classification Using Deep Learning

Time-series classification is utilized in a variety of applications leading to the development of many data mining techniques for time-series analysis. Among the broad range of time-series classification algorithms, recent studies are considering the impact of deep learning methods on time-series classification tasks. The quantity of related publications requires a bibliometric study to explore most prominent keywords, countries, sources and research clusters. The paper conducts a bibliometric analysis on related publications in time-series classification, adopted from Scopus database between 2010 and 2019. Through keywords co-occurrence analysis, a visual network structure of top keywords in time-series classification research has been produced and deep learning has been introduced as the most common topic by additional inquiry of the bibliography. The paper continues by exploring the publication trends of recent deep learning approaches for time-series classification. The annual number of publications, the productive and collaborative countries, the growth rate of sources, the most occurred keywords and the research collaborations are revealed from the bibliometric analysis within the study period. The research field has been broken down into three main categories as different frameworks of deep neural networks, different applications in remote sensing and also in signal processing for time-series classification tasks. The qualitative analysis highlights the categories of top citation rate papers by describing them in details.

Evolving CNN-LSTM Models for Time Series Prediction Using Enhanced Grey Wolf Optimizer

In this research, we propose an enhanced Grey Wolf Optimizer (GWO) for designing the evolving Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) networks for time series analysis. To overcome the probability of stagnation at local optima and a slow convergence rate of the classical GWO algorithm, the newly proposed variant incorporates four distinctive search mechanisms. They comprise a nonlinear exploration scheme for dynamic search territory adjustment, a chaotic leadership dispatching strategy among the dominant wolves, a rectified spiral local exploitation action, as well as probability distribution-based leader enhancement. The evolving CNN-LSTM models are subsequently devised using the proposed GWO variant, where the network topology and learning hyperparameters are optimized for time series prediction and classification tasks. Evaluated using a number of benchmark problems, the proposed GWO-optimized CNN-LSTM models produce statistically significant results over those from several classical search methods and advanced GWO and Particle Swarm Optimization variants. Comparing with the baseline methods, the CNN-LSTM networks devised by the proposed GWO variant offer better representational capacities to not only capture the vital feature interactions, but also encapsulate the sophisticated dependencies in complex temporal contexts for undertaking time-series tasks.

Independent Random Recurrent Neural Networks for Infrared Spatial Point Targets Classification

Exo-atmospheric infrared (IR) point target discrimination is an important research topic of space surveillance systems. It is difficult to describe the characteristic information of the shape and micro-motion states of the targets and to discriminate different targets effectively by the characteristic information. This paper has constructed the infrared signature model of spatial point targets and obtained the infrared radiation intensity sequences dataset of different types of targets. This paper aims to design an algorithm for the classification problem of infrared radiation intensity sequences of spatial point targets. Recurrent neural networks (RNNs) are widely used in time series classification tasks, but face several problems such as gradient vanishing and explosion, etc. In view of shortcomings of RNNs, this paper proposes an independent random recurrent neural network (IRRNN) model, which combines independent structure RNNs with randomly weighted RNNs. Without increasing the training complexity of network learning, our model solves the problem of gradient vanishing and explosion, improves the ability to process long sequences, and enhances the comprehensive classification performance of the algorithm effectively. Experiments show that the IRRNN algorithm performs well in classification tasks and is robust to noise.

Regularizing Fully Convolutional Networks for Time Series Classification by Decorrelating Filters

Deep neural networks are prone to overfitting, especially in small training data regimes. Often, these networks are overparameterized and the resulting learned weights tend to have strong correlations. However, convolutional networks in general, and fully convolution neural networks (FCNs) in particular, have been shown to be relatively parameter efficient, and have recently been successfully applied to time series classification tasks. In this paper, we investigate the application of different regularizers on the correlation between the learned convolutional filters in FCNs using Batch Normalization (BN) as a regularizer for time series classification (TSC) tasks. Results demonstrate that despite orthogonal initialization of the filters, the average correlation across filters (especially for filters in higher layers) tends to increase as training proceeds, indicating redundancy of filters. To mitigate this redundancy, we propose a strong regularizer, using simple yet effective filter decorrelation. Our proposed method yields significant gains in classification accuracy for 44 diverse time series datasets from the UCR TSC benchmark repository.

A deep learning framework for time series classification using Relative Position Matrix and Convolutional Neural Network

Time series classification (TSC) which has attracted great attention in time series data mining task, has already applied to various fields. With the rapid development of Convolutional Neural Network (CNN), the CNN based methods on TSC have begun to emerge until recently. However, the performance of CNN based methods is slightly worse than state-of-the-art traditional methods. Therefore, we propose a novel deep learning framework using Relative Position Matrix and Convolutional Neural Network (RPMCNN) for the TSC task. We investigate a time series data representation method called Relative Position Matrix (RPM) to convert the raw time series data to 2D images which enable the use of techniques from image recognition. We also construct an improved CNN architecture to automatically learn a high-level abstract representation of low-level raw time series data. Therefore, the combination of RPM and CNN in a unified framework is expected to boost the accuracy and generalization ability of TSC. We conduct a comprehensive evaluation with various existing methods on a large number of standard datasets and demonstrate that our approach achieves remarkable results and outperforms the current best TSC approaches by a large margin.

O-41 The role of warping window size in case of EEG classification

Background Classification of EEG signals is the common theoretical background of various EEG-related recognition tasks, such as the recognition of symptoms of diseases. We consider these tasks as time series classification tasks for which models based on dynamic time warping (DTW) are popular and effective ( Dau et al., 2018 , Buza et al., 2015 ). According to a recent study ( Dau et al., 2018 ), setting the appropriate warping window size (WWS) is crucial for the accuracy in various applications. Materials and methods Using a publicly available EEG dataset ( https://archive.ics.uci.edu/ml/datasets/eeg+database ), we examined whether the WWS is crucial in case of EEG classification. We considered two DTW-based methods, in particular, the wide-spread nearest neighbor and a more advanced approach, PROCESS ( Buza et al., 2015 ). We performed disease recognition experiments according to patient-based 10-fold cross-validation and measured the area under receiver operator characteristic curve (AUC) for both models with various WWS. Results A reasonably high accuracy is achieved in a relatively wide rage of WWS. Most importantly, WWS should not be set to zero. The AUC of the best examined model was around 0.875 which we achieved with relatively low WWS corresponding to approximately 170 ms. Conclusions Our observations show that in case of EEG classification, the examined classifiers are much less sensitive to the WWS than suggested by Dau et al. (2018) .

Time series classification with Echo Memory Networks

Echo state networks (ESNs) are randomly connected recurrent neural networks (RNNs) that can be used as a temporal kernel for modeling time series data, and have been successfully applied on time series prediction tasks. Recently, ESNs have been applied to time series classification (TSC) tasks. However, previous ESN-based classifiers involve either training the model by predicting the next item of a sequence, or predicting the class label at each time step. The former is essentially a predictive model adapted from time series prediction work, rather than a model designed specifically for the classification task. The latter approach only considers local patterns at each time step and then averages over the classifications. Hence, rather than selecting the most discriminating sections of the time series, this approach will incorporate non-discriminative information into the classification, reducing accuracy. In this paper, we propose a novel end-to-end framework called the Echo Memory Network (EMN) in which the time series dynamics and multi-scale discriminative features are efficiently learned from an unrolled echo memory using multi-scale convolution and max-over-time pooling. First, the time series data are projected into the high dimensional nonlinear space of the reservoir and the echo states are collected into the echo memory matrix, followed by a single multi-scale convolutional layer to extract multi-scale features from the echo memory matrix. Max-over-time pooling is used to maintain temporal invariance and select the most important local patterns. Finally, a fully-connected hidden layer feeds into a softmax layer for classification. This architecture is applied to both time series classification and human action recognition datasets. For the human action recognition datasets, we divide the action data into five different components of the human body, and propose two spatial information fusion strategies to integrate the spatial information over them. With one training-free recurrent layer and only one layer of convolution, the EMN is a very efficient end-to-end model, and ranks first in overall classification ability on 55 TSC benchmark datasets and four 3D skeleton-based human action recognition tasks.

Multivariate LSTM-FCNs for time series classification

Over the past decade, multivariate time series classification has received great attention. We propose transforming the existing univariate time series classification models, the Long Short Term Memory Fully Convolutional Network (LSTM-FCN) and Attention LSTM-FCN (ALSTM-FCN), into a multivariate time series classification model by augmenting the fully convolutional block with a squeeze-and-excitation block to further improve accuracy. Our proposed models outperform most state-of-the-art models while requiring minimum preprocessing. The proposed models work efficiently on various complex multivariate time series classification tasks such as activity recognition or action recognition. Furthermore, the proposed models are highly efficient at test time and small enough to deploy on memory constrained systems.

Spiking Echo State Convolutional Neural Network for Robust Time Series Classification

In this paper, a novel high-accuracy and robust computing framework for time series classification tasks is presented. The framework consists of a feature extraction module and a classification module, where the feature extraction is implemented by reservoir computing method of spiking neurons, and the classification result is obtained by the state-of-the-art analog convolutional neural networks (CNNs). The original time series input is first converted to multi-channel spike streams, then fed into the spiking reservoir layer to produce intermediate spike output, and subsequently, the spike output is transformed into a 2D mapping image, and deep CNN model is applied to classify the mapping image. The proposed model has the following three significant advantages: long-and-short term memory brought by the echo state of reservoir component, robustness to noise brought by the spiking encoding method, and high-accuracy performance brought by the deep CNN model. The experiments conducted on both synthetic time series data set and UCR time series data sets showed that our approach achieved highly competitive accuracy and robustness over other existing methods.

Dynamic Barycenter Averaging Kernel in RBF Networks for Time Series Classification

Radial basis function (RBF) network has been utilized in many applications due to its simple topological structure and strong capacity on function approximation. The core of RBF network is its static kernel function, which is based on the Euclidean distance and cannot obtain good performance for time series classification (TSC) due to the time-shift invariance, complex dynamics, and different length of temporal data.This paper proposed a new temporal kernel, namely, the dynamic barycenter averaging kernel (DBAK) and introduced it into the RBF network. First, we combine k-means clustering with a dynamic time warping (DTW)-based averaging algorithm called DTW barycenter averaging (DBA) to determine the center of DBAK. Then, in order to facilitate the stable gradient-training process in the whole network, a normalization term is added into the kernel formulation. By integrating the information of the whole time warping path, our DBAK-based RBF network performs efficiently for TSC tasks.

Recurrent neural networks for discrimination of exo-atmospheric targets based on infrared radiation signature

Exo-atmospheric infrared (IR) target discrimination is an important research problem in space attack and defense. The different micro-motion states of the targets result in respective characteristics in the obtained IR radiation intensity sequences, and this difference is difficult to describe intuitively and extract effectively. Few methods can effectively contact the data with the micro-motion model, resulting in a low classification accuracy and difficult to meeting actual application requirements. We set up four types of targets by constructing the micro-motion model of the exo-atmospheric targets, including the warhead and heavy decoy, with spinning and coning motion; and the light decoy and debris, with tumbling motion, to get the IR radiation intensity sequences. We use random projection to improve the discrimination power of recurrent neural network, and to classify the time series of IR radiation intensity. Experimental results demonstrate that random projection recurrent neural network (R-RNN) is more effective than several other typical algorithms in time series classification (TSC) task, which can achieve an excellent target discrimination. We also analyze the effect of noise on the performance of the algorithm.

Reservoir computing with a single delay-coupled non-linear mechanical oscillator

Reservoir computing was achieved by constructing a network of virtual nodes multiplexed in time and sharing a single silicon beam exhibiting a classical Duffing non-linearity as the source of nonlinearity. The delay-coupled electromechanical system performed well on time series classification tasks, with error rates below 0.1% for the 1st, 2nd, and 3rd order parity benchmarks and an accuracy of (78±2)% for the TI-46 spoken word recognition benchmark. As a first demonstration of reservoir computing using a non-linear mass-spring system in MEMS, this result paves the way to the creation of a new class of compact devices combining the functions of sensing and computing.

A Time-Distributed Spatiotemporal Feature Learning Method for Machine Health Monitoring with Multi-Sensor Time Series.

Data-driven methods with multi-sensor time series data are the most promising approaches for monitoring machine health. Extracting fault-sensitive features from multi-sensor time series is a daunting task for both traditional data-driven methods and current deep learning models. A novel hybrid end-to-end deep learning framework named Time-distributed ConvLSTM model (TDConvLSTM) is proposed in the paper for machine health monitoring, which works directly on raw multi-sensor time series. In TDConvLSTM, the normalized multi-sensor data is first segmented into a collection of subsequences by a sliding window along the temporal dimension. Time-distributed local feature extractors are simultaneously applied to each subsequence to extract local spatiotemporal features. Then a holistic ConvLSTM layer is designed to extract holistic spatiotemporal features between subsequences. At last, a fully-connected layer and a supervised learning layer are stacked on the top of the model to obtain the target. TDConvLSTM can extract spatiotemporal features on different time scales without any handcrafted feature engineering. The proposed model can achieve better performance in both time series classification tasks and regression prediction tasks than some state-of-the-art models, which has been verified in the gearbox fault diagnosis experiment and the tool wear prediction experiment.

Recurrent Neural Networks for Multivariate Time Series with Missing Values

Multivariate time series data in practical applications, such as health care, geoscience, and biology, are characterized by a variety of missing values. In time series prediction and other related tasks, it has been noted that missing values and their missing patterns are often correlated with the target labels, a.k.a., informative missingness. There is very limited work on exploiting the missing patterns for effective imputation and improving prediction performance. In this paper, we develop novel deep learning models, namely GRU-D, as one of the early attempts. GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the-art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture so that it not only captures the long-term temporal dependencies in time series, but also utilizes the missing patterns to achieve better prediction results. Experiments of time series classification tasks on real-world clinical datasets (MIMIC-III, PhysioNet) and synthetic datasets demonstrate that our models achieve state-of-the-art performance and provide useful insights for better understanding and utilization of missing values in time series analysis.