Multivariate Time Series Datasets Research Articles

SCMFTS: Scalable and Distributed Complexity Measures and Features for Univariate and Multivariate Time Series in Big Data Environments

Time series data are becoming increasingly important due to the interconnectedness of the world. Classical problems, which are getting bigger and bigger, require more and more resources for their processing, and Big Data technologies offer many solutions. Although the principal algorithms for traditional vector-based problems are available in Big Data environments, the lack of tools for time series processing in these environments needs to be addressed. In this work, we propose a scalable and distributed time series transformation for Big Data environments based on well-known time series features (SCMFTS), which allows practitioners to apply traditional vector-based algorithms to time series problems. The proposed transformation, along with the algorithms available in Spark, improved the best results in the state-of-the-art on the Wearable Stress and Affect Detection dataset, which is the biggest publicly available multivariate time series dataset in the University of California Irvine (UCI) Machine Learning Repository. In addition, SCMFTS showed a linear relationship between its runtime and the number of processed time series, demonstrating a linear scalable behavior, which is mandatory in Big Data environments. SCMFTS has been implemented in the Scala programming language for the Apache Spark framework, and the code is publicly available.

Multivariate Time Series Data Prediction Based on ATT-LSTM Network

Deep learning models have been widely used in prediction problems in various scenarios and have shown excellent prediction effects. As a deep learning model, the long short-term memory neural network (LSTM) is potent in predicting time series data. However, with the advancement of technology, data collection has become more accessible, and multivariate time series data have emerged. Multivariate time series data are often characterized by a large amount of data, tight timeline, and many related sequences. Especially in real data sets, the change rules of many sequences will be affected by the changes of other sequences. The interacting factors data, mutation information, and other issues seriously impact the prediction accuracy of deep learning models when predicting this type of data. On the other hand, we can also extract the mutual influence information between different sequences and simultaneously use the extracted information as part of the model input to make the prediction results more accurate. Therefore, we propose an ATT-LSTM model. The network applies the attention mechanism (attention) to the LSTM to filter the mutual influence information in the data when predicting the multivariate time series data, which makes up for the poor ability of the network to process data. Weaknesses have greatly improved the accuracy of the network in predicting multivariate time series data. To evaluate the model’s accuracy, we compare the ATT-LSTM model with the other six models on two real multivariate time series data sets based on two evaluation indicators: Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). The experimental results show that the model has an excellent performance improvement compared with the other six models, proving the model’s effectiveness in predicting multivariate time series data.

Tree-tensor-network classifiers for machine learning: From quantum inspired to quantum assisted

We describe a quantum-assisted machine learning (QAML) method in which multivariate data is encoded into quantum states in a Hilbert space whose dimension is exponentially large in the length of the data vector. Learning in this space occurs through applying a low-depth quantum circuit with a tree tensor network (TTN) topology, which acts as an unsupervised feature extractor to identify the most relevant quantum states in a data-driven fashion. This unsupervised feature extractor then feeds a supervised linear classifier and encodes the output in a small-dimensional quantum register. In contrast to previous work on \emph{quantum-inspired} TTN classifiers, in which the embedding map and class decision weights did not map the data to well-defined quantum states, we present an approach that can be implemented on gate-based quantum computing devices. In particular, we identify an embedding map with accuracy similar to exponential machines (Novikov \emph{et al.}, arXiv:1605.03795), but which produces valid quantum states from classical data vectors, and utilize manifold-based gradient optimization schemes to produce isometric operations mapping quantum states to a register of qubits defining a class decision. We detail methods for efficiently obtaining one- and two-point correlation functions of the decision boundary vectors of the quantum model, which can be used for model interpretability, as well as methods for obtaining classifications from partial data vectors. Further, we show that the use of isometric tensors can significantly aid in the human interpretability of the correlation functions extracted from the decision weights, and may produce models that are less susceptible to adversarial perturbations. We demonstrate our methodologies in applications utilizing the MNIST handwritten digit dataset and a multivariate timeseries dataset of human activity recognition.

Outlier detection for multivariate time series: A functional data approach

A method for detecting outlier samples in a multivariate time series dataset is proposed. It is assumed that an outlying series is characterized by having been generated from a different process than those associated with the rest of the series. Each multivariate time series is described by means of an estimator of its quantile cross-spectral density, which is treated as a multivariate functional datum. Then an outlier score is assigned to each series by using functional depths. A broad simulation study shows that the proposed approach is superior to the alternatives suggested in the literature and demonstrates that the consideration of functional data constitutes a critical step. The procedure runs in linear time with respect to both the series length and the number of series, and in quadratic time with respect to the number of dimensions. Two applications concerning financial series and ECG signals highlight the usefulness of the technique.

Robust Graph Factorization for Multivariate Electricity Consumption Series Clustering

Multivariate electricity consumption series clustering can reflect trends of power consumption changes in the past time period, which can provide reliable guidance for electricity production. However, there are some abnormal series in the past multivariate electricity consumption series data, while outliers will affect the discovery of electricity consumption trends in different time periods. To address this problem, we propose a robust graph factorization model for multivariate electricity consumption clustering (RGF-MEC), which performs graph factorization and outlier discovery simultaneously. RGF-MEC first obtains a similarity graph by calculating distance among multivariate electricity consumption series data and then performs robust matrix factorization on the similarity graph. Meanwhile, the similarity graph is decomposed into a class-related embedding and a spectral embedding, where the class-related embedding directly reveals the final clustering results. Experimental results on realistic multivariate time-series datasets and multivariate electricity consumption series datasets demonstrate effectiveness of the proposed RGF-MEC model.

Quantitative Confocal Microscopy for Grouping of Dose-Response Data: Deciphering Calcium Sequestration and Subsequent Cell Death in the Presence of Excess Norepinephrine.

Fluorescent calcium (Ca2+) imaging is one of the preferred methods to record cellular activity during in vitro preclinical studies, high-content drug screening, and toxicity analysis. Visualization and analysis for dose-response data obtained using high-resolution imaging remain challenging, due to the inherent heterogeneity present in the Ca2+ spiking. To address this challenge, we propose measurement of cytosolic Ca2+ ions using spinning-disk confocal microscopy and machine learning-based analytics that is scalable. First, we implemented uniform manifold approximation and projection (UMAP) for visualizing the multivariate time-series dataset in the two-dimensional (2D) plane using Python. The dataset was obtained through live imaging experiments with norepinephrine-induced Ca2+ oscillation in HeLa cells for a large range of doses. Second, we demonstrate that the proposed framework can be used to depict the grouping of the spiking pattern for lower and higher drug doses. To the best of our knowledge, this is the first attempt at UMAP visualization of the time-series dose response and identification of the Ca2+ signature during lytic death. Such quantitative microscopy can be used as a component of a high-throughput data analysis workflow for toxicity analysis.

Learnable Dynamic Temporal Pooling for Time Series Classification

With the increase of available time series data, predicting their class labels has been one of the most important challenges in a wide range of disciplines. Recent studies on time series classification show that convolutional neural networks (CNN) achieved the state-of-the-art performance as a single classifier. In this work, pointing out that the global pooling layer that is usually adopted by existing CNN classifiers discards the temporal information of high-level features, we present a dynamic temporal pooling (DTP) technique that reduces the temporal size of hidden representations by aggregating the features at the segment-level. For the partition of a whole series into multiple segments, we utilize dynamic time warping (DTW) to align each time point in a temporal order with the prototypical features of the segments, which can be optimized simultaneously with the network parameters of CNN classifiers. The DTP layer combined with a fully-connected layer helps to extract further discriminative features considering their temporal position within an input time series. Extensive experiments on both univariate and multivariate time series datasets show that our proposed pooling significantly improves the classification performance.

Reservoir Computing Approaches for Representation and Classification of Multivariate Time Series.

Classification of multivariate time series (MTS) has been tackled with a large variety of methodologies and applied to a wide range of scenarios. Reservoir computing (RC) provides efficient tools to generate a vectorial, fixed-size representation of the MTS that can be further processed by standard classifiers. Despite their unrivaled training speed, MTS classifiers based on a standard RC architecture fail to achieve the same accuracy of fully trainable neural networks. In this article, we introduce the reservoir model space, an unsupervised approach based on RC to learn vectorial representations of MTS. Each MTS is encoded within the parameters of a linear model trained to predict a low-dimensional embedding of the reservoir dynamics. Compared with other RC methods, our model space yields better representations and attains comparable computational performance due to an intermediate dimensionality reduction procedure. As a second contribution, we propose a modular RC framework for MTS classification, with an associated open-source Python library. The framework provides different modules to seamlessly implement advanced RC architectures. The architectures are compared with other MTS classifiers, including deep learning models and time series kernels. Results obtained on the benchmark and real-world MTS data sets show that RC classifiers are dramatically faster and, when implemented using our proposed representation, also achieve superior classification accuracy.

Multi-task learning based Encoder-Decoder: A comprehensive detection and diagnosis system for multi-sensor data

As more and more sensor data have been collected, automated detection, and diagnosis systems are urgently needed to lessen the increasing monitoring burden and reduce the risk of system faults. A plethora of researches have been done on anomaly detection, event detection, anomaly diagnosis respectively. However, none of current approaches can explore all these respects in one unified framework. In this work, a Multi-Task Learning based Encoder-Decoder (MTLED) which can simultaneously detect anomalies, diagnose anomalies, and detect events is proposed. In MTLED, feature matrix is introduced so that features are extracted for each time point and point-wise anomaly detection can be realized in an end-to-end way. Anomaly diagnosis and event detection share the same feature matrix with anomaly detection in the multi-task learning framework and also provide important information for system monitoring. To train such a comprehensive detection and diagnosis system, a large-scale multivariate time series dataset which contains anomalies of multiple types is generated with simulation tools. Extensive experiments on the synthetic dataset verify the effectiveness of MTLED and its multi-task learning framework, and the evaluation on a real-world dataset demonstrates that MTLED can be used in other application scenarios through transfer learning.

Identify Significant Phenomenon-Specific Variables for Multivariate Time Series

Multivariate time series (MTS) are collected for different variables in studying scientific phenomena or monitoring system health where each time series records the values of one variable for a time period. Among the different variables, it is common that only a few variables contribute significantly to a specific phenomenon. Furthermore, the variables contributing significantly to different phenomena are often different. We denote the different variables that contribute to the occurrences of different phenomena as Phenomenon-specific Variables (PVs) . In this paper, we formulate a novel problem of identifying significant PVs from MTS datasets. To analyze MTS data, feature extraction techniques have been extensively studied. However, most of them identify important global features for one dataset and do not utilize the temporal order of time series. To solve the newly introduced problem, we propose a solution framework, CNN $_{mts}$ m t s -X , which is a new variant of the Convolutional Neural Networks ( CNN ) and can embed other feature extraction techniques (as X ). Furthermore, we design a CNN $_{mts}$ m t s -LR method that implements a new feature identification approach ( LR ) as X in the CNN $_{mts}$ m t s -X framework. The LR method leverages both Linear Discriminant Analysis ( LDA ) and Random Forest ( RF ). Our extensive experiments on five real datasets show that the CNN $_{mts}$ m t s -LR method has exhibited much better performance than several other baseline methods. Using 30 percent of the PVs discovered from the CNN $_{mts}$ m t s -LR , classifications can achieve better or similar performance than using all the variables.

Outlier Detection for Multivariate Time Series Using Dynamic Bayesian Networks

Outliers are observations suspected of not having been generated by the underlying process of the remaining data. Many applications require a way of identifying interesting or unusual patterns in multivariate time series (MTS), now ubiquitous in many applications; however, most outlier detection methods focus solely on univariate series. We propose a complete and automatic outlier detection system covering the pre-processing of MTS data that adopts a dynamic Bayesian network (DBN) modeling algorithm. The latter encodes optimal inter and intra-time slice connectivity of transition networks capable of capturing conditional dependencies in MTS datasets. A sliding window mechanism is employed to score each MTS transition gradually, given the DBN model. Two score-analysis strategies are studied to assure an automatic classification of anomalous data. The proposed approach is first validated in simulated data, demonstrating the performance of the system. Further experiments are made on real data, by uncovering anomalies in distinct scenarios such as electrocardiogram series, mortality rate data, and written pen digits. The developed system proved beneficial in capturing unusual data resulting from temporal contexts, being suitable for any MTS scenario. A widely accessible web application employing the complete system is publicly available jointly with a tutorial.

Multivariate time-series clustering based on component relationship networks

Clustering is a powerful technique for providing class labels of data objects for learning guidance. Because traditional clustering methods rarely consider the component correlations of a multivariate time series(MTS), an MTS clustering method based on a component relationship network (CRN) is proposed in the present study. An MTS dataset is mapped to a multi-relationship network (MRN), which consists of a set of CRNs. Every CRN reflects the relationship of the MTS data under each component. The proposed method applies two steps. First, the distance function and K-nearest neighbors are combined to transform an MTS dataset into an MRN. Second, a non-negative matrix factorization is designed to identify time-series clusters. Because asynchronous time-series data exist in the form of an approximate case, we recommend an improved penalty-coefficient based dynamic time-warping algorithm to measure the similarity between two-time sequences. The similarity can reflect the correlation between the asynchronous MTS data. Through an experiment, we compared the proposed method to other clustering methods and discussed of the effects of the parameters in detail. The numerical results of the experiment demonstrate that the proposed method can improve the accuracy and quality of MTS data clustering.

Multivariate Time Series Forecasting with Dilated Residual Convolutional Neural Networks for Urban Air Quality Prediction

Multivariate time series (MTS) forecasting is a research field that is gaining more and more importance as time series data generators proliferate in the growing era of Internet of Things. Deep learning architecture for MTS data has been and still a very active research area as there is no comprehensive comparative study of the different architectures, let alone a perfect architecture that can solve all types of time series modeling problem. In this paper, we start by highlighting time series analysis requirements, to propose a new model for conditional multivariate time series forecasting based on the lately introduced WaveNet architecture. Our model is composed of stacked residual causal dilated convolutions that provide large scope in time series history and foster learning of long-term dependencies. Parameterized skip connections allow catching early trends and properties while conditioning enables modeling the associations that exist between multivariate data. We used group normalization for its stability and independence from the batch size. We present a deep comparison of the structure, flexibility, flow control, memory consumption, robustness and stability capacities of our new model against those of recurrent neural networks state-of-the-art approaches—long-short-term-memory (LSTM) and gated recurrent unit (GRU). To assess the performances of our model, we conduct extensive experimental testing on the task of urban air quality prediction in Marrakesh city using six real-world multi-sensor and multivariate time series datasets. We compare the results of our model against the results of LSTM and GRU. Our experiments revealed that our proposed WaveNet-temporal-CNN outperform recurrent models ability to learn long-term dependencies in a time-efficient way.

Multivariate Extreme Learning Machine Based AutoEncoder for Electricity Consumption Series Clustering

Multivariate electricity consumption series clustering can reflect the trend of power consumption changes in the past time period, which can provide reliable guidance for electricity production. The dimensionality reduction-based method is an effective technology to address this problem, which obtains the low-dimensional features of each variate or all variates for multivariate time series clustering. However, most existing dimensionality reduction-based methods ignore the joint learning of the common representations and the variable-based representations. In this paper, we build a multivariate extreme learning machine based autoencoder model for electricity consumption clustering (MELM-EC), which performs common representation learning and variable-based representation learning simultaneously. MELM-EC maps the common representation and multiple variable-based representations to the original multivariate time series and computes the common output weights within a few iterations. Experimental results on realistic multivariate time series datasets and multivariate electricity consumption series datasets demonstrate the effectiveness of the proposed MELM-EC model.

Pseudo Bidirectional Linear Discriminant Analysis for Multivariate Time Series Classification

Multivariate time series (MTS) is a kind of matrix data, typically consisting of multiple variables measured at multiple time points. Due to the high dimensionality of MTS data, many methods for MTS classification have been proposed within the literature to reduce the redundancy in time or variable mode, but there is relatively little work on exploring the redundancy in both modes concurrently. In this paper we propose a new method for MTS classification based on bidirectional linear discriminant analysis (BLDA). The advantage is that BLDA can utilize label information in reducing the redundancy in time and variable modes simultaneously. Moreover, the existing procedures for BLDA suffer from two problems: (i) BLDA cannot be performed when one of the within-class matrices is singular; (ii) the computational burden could be very heavy when one of the data dimensionality is high. A new procedure for BLDA based on pseudo-inverse (PBLDA) and an efficient algorithm for PBLDA are proposed in this paper to overcome the two problems. The performance of our proposed method is illustrated through the experiments on a number of real MTS datasets.

Analysis of energy consumption using RNN-LSTM and ARIMA Model

Given the increase of smart electricity meters and the wide adoption of electricity generation technologies such as solar panels, there is a wealth of data available on the usage of electricity. This data represents a multivariate time series of variables related to power, which could in turn be used to model and even forecast future electricity usage. The Household Power Consumption dataset is a multivariate time series dataset which describes the electricity consumption over four years for a single household. They were tested to predict for a specific house and block of houses over a given period of time. Throughout the past couple of decades energy demand has increased exponentially. This increase loads the electricity distributors heavily. So forecasting future demand for electricity use would give the dealer an upper hand. Predicting the consumption of energy requires several parameters. This paper proposes two methods with one using a Recurrent Neural Network (RNN) and another using a Long Short Term Memory (LSTM) network, considering only the previous consumption of electricity to estimate potential consumption of electricity. To assess the applicability of the RNN and the LSTM network to predict the electricity consumption.

Missing value imputation in multivariate time series with end-to-end generative adversarial networks

Missing values are inherent in multivariate time series because of multiple reasons, such as collection errors, which deteriorate the performance of follow-up analytic applications on the multivariate time series. Numerous missing value imputation methods have been proposed to mitigate the influence of missing values on multivariate time series analysis. Recently, inspired by the success of generative adversarial networks (GANs) in image generation, the GAN-2-Stage has been used to address the imputation problem with the generative model. Specifically, GAN-2-Stage employs GANs to impute the missing values. However, an extra phase is required to optimize the input random “noise” of the generator. In addition, the imputed values can be very different from real values because of the difficulty in training a GAN and the unstable generation process. Therefore, this paper proposes an end-to-end model to impute the missing values in a multivariate time series. Specifically, we introduce an encoder network into the standard GAN architecture that eliminates the input optimization phase in the GAN-2-Stage. Our generator utilizes real data during training to force the imputed values to be close to the real ones. Experiments on three real-world multivariate time series datasets demonstrate that the proposed model outperforms state-of-the-art methods in imputation tasks and downstream applications, including classification and regression.

Deep learning for classification of time series spectral images using combined multi-temporal and spectral features

Time series spectral imaging facilitates a comprehensive understanding of the underlying dynamics of multi-component systems and processes. Most existing classification strategies focus exclusively on the spectral features and they tend to fail when spectra between classes closely resemble each other. This work proposes a hybrid approach of principal component analysis (PCA) and deep learning (i.e., long short-term memory (LSTM) model) for incorporating and utilizing the combined multi-temporal and spectral information from time series spectral imaging datasets. An example data, consisting of times series spectral images of casein-based biopolymers, was used to illustrate and evaluate the proposed hybrid approach. Compared to using partial least squares discriminant analysis (PLSDA), the proposed PCA-LSTM method applying the same spectral pretreatment achieved substantial improvement in the pixel-wise classification (i.e., accuracy increased from 59.97% of PLSDA to 85.73% of PCA-LSTM). When projecting the pixel-wise model to object-based classification, the PCA-LSTM approach produced an accuracy of 100%, correctly classifying the whole 21 film samples in the independent test set, while PLSDA only led to an accuracy of 80.95%. The proposed method is powerful and versatile in utilizing distinctive characteristics of time dependencies from multivariate time series dataset, which could be adapted to suit non-congruent images over time sequences as well as spectroscopic data.

A deep gated recurrent neural network for petroleum production forecasting

Forecasting of oil production plays a vital role in petroleum engineering and contributes to supporting engineers in the management of petroleum reservoirs. However, reliable production forecasting is difficult to achieve, particularly in view of the increase in digital oil big data. Although a significant amount of work has been reported in the literature in relation to the use of machine learning in the oil and gas domain, traditional forecasting approaches have limited potential in terms of representing the complex features of time series data. More specifically, in a high-dimensional nonlinear multivariate time series dataset, a shallow machine is incapable of inferring the dependencies between past and future values. In this context, a novel forecasting model for petroleum production is proposed in this work. The model is a deep-gated recurrent neural network consisting of multiple hidden layers, where each layer has a number of nodes. The proposed model has a low-complexity architecture and the capacity to track long-interval time-series datasets. To evaluate the robustness of our model, the proposed technique was benchmarked with various standard approaches. The extensive empirical results demonstrate that the proposed model outperforms existing approaches.

Process Monitoring and Fault Prediction in Multivariate Time Series Using Bag-of-Words

Multivariate time series (MTS) arise due to multisensor data collection in manufacturing. These data are complex in the sense that attributes have a varying scale, volitivity, continuity, and so on, and interattribute dependence also appears, which can mask the inherent information about system health status. Conventional machine learning-based process monitoring techniques are inefficient in predicting faults with MTS—their detection capability heavily relies on the input features, yet the classification power of MTS attributes is weakened by the complexity of multisensor data. Effective feature extraction is, therefore, necessary to facilitate fault prediction with MTS. This study proposes a fault prediction framework for MTS based on bag-of-words (BOW) feature extraction, statistical feature selection, and classification analysis. BOW models are for the first time adopted in a manufacturing context. Their superior capability in information preservation, local pattern recognition, and temporal effect accommodation has overcome the major limitations in current manufacturing practices with MTS. A comprehensive case study demonstrates the desirable performance of this framework on two MTS data sets from paper manufacturing and automotive manufacturing, as well as its superiority over conventional machine learning-based fault prediction. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Process monitoring in a multisensor environment has been a vital interest in manufacturing. The difficulty lies in the lack of detection power in many conventional techniques, e.g., control chart and logistic regression. A critical reason for such failure is the neglect of time effect in MTS—patterns associated with system faults tend to stretch a period, but a conventional control chart or classifier inspects each time stamp separately. Fault detection based on time series sequences is, therefore, essential. However, how to effectively extract features from MTS becomes a challenge. The framework proposed in this study adopts BOW models, specifically symbolic aggregate approximation (SAX), to extract features from MTS sequences, thus substantially improves the detection power against local patterns over time. Many manufacturing multisensor data are subject to such local patterns that point to the root cause of fault, so the proposed framework has a wide application in manufacturing.