High-dimensional Multivariate Time Series Research Articles

Latent Diffusion Transformer for Probabilistic Time Series Forecasting

The probability prediction of multivariate time series is a notoriously challenging but practical task. This research proposes to condense high-dimensional multivariate time series forecasting into a problem of latent space time series generation, to improve the expressiveness of each timestamp and make forecasting more manageable. To solve the problem that the existing work is hard to extend to high-dimensional multivariate time series, we present a latent multivariate time series diffusion framework called Latent Diffusion Transformer (LDT), which consists of a symmetric statistics-aware autoencoder and a diffusion-based conditional generator, to implement this idea. Through careful design, the time series autoencoder can compress multivariate timestamp patterns into a concise latent representation by considering dynamic statistics. Then, the diffusion-based conditional generator is able to efficiently generate realistic multivariate timestamp values on a continuous latent space under a novel self-conditioning guidance which is modeled in a non-autoregressive way. Extensive experiments demonstrate that our model achieves state-of-the-art performance on many popular high-dimensional multivariate time series datasets.

Adaptive Wavelet Domain Principal Component Analysis for Nonstationary Time Series

High-dimensional multivariate nonstationary time series, that is, data whose second order properties vary over time, are common in many scientific and industrial applications. In this article we propose a novel wavelet domain dimension reduction technique for nonstationary time series. By constructing a time-scale adaptive principal component analysis of the data, our proposed method is able to capture the salient dynamic features of the multivariate time series. We also introduce a new time and scale dependent cross-coherence measure to quantify the extent of association between a multivariate nonstationary time series and its proposed wavelet domain principal component representation. Theoretical results establish that our associated estimation scheme enjoys good bias and consistency properties when determining wavelet domain principal components of input data. The proposed method is illustrated using extensive simulations and we demonstrate its applicability on a real-world dataset arising in a neuroscience study. Supplementary materials, with proofs of theoretical results, additional simulations and code, are available online.

Information-based estimation of causality networks from high-dimensional multivariate time series

Abstract One of the most challenging aspects in the study of the complex dynamical systems is the estimation of their underlying, interdependence structure. Being in the era of Big Data, this problem gets even more complicated since more observed variables are available. To estimate direct causality effects in this setting, dimension reduction has to be employed in the Granger causality measure. The measure should also be capable to detect non-linear effects, persistently present in real-world complex systems. The model-free information-based measure of partial mutual information from mixed embedding (PMIME) has been developed to address these issues and it was found to perform well on multivariate time series of moderately high dimension. Here, the problem of forming complex networks from direct, possibly non-linear, high-dimensional time series at the order of hundreds is investigated. The performance of the measure PMIME is tested on two coupled dynamical systems in discrete time (coupled Hénon maps) and continuous time (coupled Mackey–Glass delay differential equations). It is concluded that the correct detection of the underlying causality network depends mainly on the network density rather than on its size (number of nodes). Finally, the effect of network size is investigated in the study of the British stock market in the period around Brexit.

A Review of Outlier Detection and Robust Estimation Methods for High Dimensional Time Series Data

Diagnostic procedures for finding outliers in high dimensional multivariate time series and robust estimation methods for these data are reviewed. First, methods for searching for outliers assuming that the data have been generated by a Dynamic Factor Model are presented. Then, other existing methods for detecting different types of multivariate time series outliers are analyzed. They include identifying outlying series and, also, looking for segments, or periods of time, where the series have unusual dynamics. Second, robust estimation methods are considered. Dynamic Principal Components, as a very general procedure to estimate the dynamic in a high dimensional data set, is introduced and different types of robust estimation of these components are reviewed. Dynamic Principal Components can be applied for robust estimation of Generalized Dynamic Factor models and some results are given. Finally, other methods proposed for robust estimation of high dimensional VAR models and other multivariate time series problems are also briefly discussed.

Deep belief improved bidirectional LSTM for multivariate time series forecasting.

Multivariate time series (MTS) play essential roles in daily life because most real-world time series datasets are multivariate and rich in time-dependent information. Traditional forecasting methods for MTS are time-consuming and filled with complicated limitations. One efficient method being explored within the dynamical systems is the extended short-term memory networks (LSTMs). However, existing MTS models only partially use the hidden spatial relationship as effectively as LSTMs. Shallow LSTMs are inadequate in extracting features from high-dimensional MTS; however, the multilayer bidirectional LSTM (BiLSTM) can learn more MTS features in both directions. This study tries to generate a novel and improved BiLSTM network (DBI-BiLSTM) based on a deep belief network (DBN), bidirectional propagation technique, and a chained structure. The deep structures are constructed by a DBN layer and multiple stacked BiLSTM layers, which increase the feature representation of DBI-BiLSTM and allow for the model to further learn the extended features in two directions. First, the input is processed by DBN to obtain comprehensive features. Then, the known features, divided into clusters based on a global sensitivity analysis method, are used as the inputs of every BiLSTM layer. Meanwhile, the previous outputs of the shallow layer are combined with the clustered features to reconstitute new input signals for the next deep layer. Four experimental real-world time series datasets illustrate our one-step-ahead prediction performance. The simulating results confirm that the DBI-BiLSTM not only outperforms the traditional shallow artificial neural networks (ANNs), deep LSTMs, and some recently improved LSTMs, but also learns more features of the MTS data. As compared with conventional LSTM, the percentage improvement of DBI-BiLSTM on the four MTS datasets is 85.41, 75.47, 61.66 and 30.72%, respectively.

An embedding-based non-stationary fuzzy time series method for multiple output high-dimensional multivariate time series forecasting in IoT applications

An embedding-based non-stationary fuzzy time series method for multiple output high-dimensional multivariate time series forecasting in IoT applications

An explainable knowledge distillation method with XGBoost for ICU mortality prediction

Background and Objective:Mortality prediction is an important task in intensive care unit (ICU) for quantifying the severity of patients’ physiological condition. Currently, scoring systems are widely applied for mortality prediction, while the performance is unsatisfactory in many clinical conditions due to the non-specificity and linearity characteristics of the used model. As the availability of the large volume of data recorded in electronic health records (EHRs), deep learning models have achieved state-of-art predictive performance. However, deep learning models are hard to meet the requirement of explainability in clinical conditions. Hence, an explainable Knowledge Distillation method with XGBoost (XGB-KD) is proposed to improve the predictive performance of XGBoost while supporting better explainability. Methods:In this method, we first use outperformed deep learning teacher models to learn the complex patterns hidden in high-dimensional multivariate time series data. Then, we distill knowledge from soft labels generated by the ensemble of teacher models to guide the training of XGBoost student model, whose inputs are meaningful features obtained from feature engineering. Finally, we conduct model calibration to obtain predicted probabilities reflecting the true posterior probabilities and use SHapley Additive exPlanations (SHAP) to obtain insights about the trained model. Results:We conduct comprehensive experiments on MIMIC-III dataset to evaluate our method. The results demonstrate that our method achieves better predictive performance than vanilla XGBoost, deep learning models and several state-of-art baselines from related works. Our method can also provide intuitive explanations. Conclusions:Our method is useful for improving the predictive performance of XGBoost by distilling knowledge from deep learning models and can provide meaningful explanations for predictions.

Latent adversarial regularized autoencoder for high-dimensional probabilistic time series prediction

Many practical applications require probabilistic prediction of time series to model the distribution on future horizons. With ever-increasing dimensions, much effort has been invested into developing methods that often make an assumption about the independence between time series. Consequently, the probabilistic prediction in high-dimensional environments has become an essential topic with significant challenges. In this paper, we propose a novel probabilistic model called latent adversarial regularized autoencoder, abbreviated as TimeLAR, specifically for high-dimensional multivariate Time Series Prediction (TSP). It integrates the flexibility of Generative Adversarial Networks (GANs) and the capability of autoencoders in extracting higher-level non-linear features. Through flexible autoencoder mapping, TimeLAR learns cross-series relationships and encodes this global information into several latent variables. We design a modified Transformer for these latent variables to capture global temporal patterns and infer latent space prediction distributions, where only one step is required to output multi-step predictions. Furthermore, we employ the GAN to further refine the performance of latent space predictions, by using a discriminator to guide the training of the autoencoder and the Transformer in an adversarial process. Finally, complex distributions of multivariate time series data can be modeled by the non-linear decoder of the autoencoder. The effectiveness of TimeLAR is empirically underpinned by extensive experiments conducted on five real-world high-dimensional time series datasets in the fields of transportation, electricity, and web page views.

Random projection ensemble classification with high-dimensional time series.

Multivariate time-series (MTS) data are prevalent in diverse domains and often high dimensional. We propose new random projection ensemble classifiers with high-dimensional MTS. The method first applies dimension reduction in the time domain via randomly projecting the time-series variables into some low-dimensional space, followed by measuring the disparity via some novel base classifier between the data and the candidate generating processes in the projected space. Our contributions are twofold: (i) We derive optimal weighted majority voting schemes for pooling information from the base classifiers for multiclass classification and (ii) we introduce new base frequency-domain classifiers based on Whittle likelihood (WL), Kullback-Leibler (KL) divergence, eigen-distance (ED), and Chernoff (CH) divergence. Both simulations for binary and multiclass problems, and an Electroencephalogram (EEG) application demonstrate the efficacy of the proposed methods in constructing accurate classifiers with high-dimensional MTS.

Temporal Latent Auto-Encoder: A Method for Probabilistic Multivariate Time Series Forecasting

Probabilistic forecasting of high dimensional multivariate time series is a notoriously challenging task, both in terms of computational burden and distribution modeling. Most previous work either makes simple distribution assumptions or abandons modeling cross-series correlations. A promising line of work exploits scalable matrix factorization for latent-space forecasting, but is limited to linear embeddings, unable to model distributions, and not trainable end-to-end when using deep learning forecasting. We introduce a novel temporal latent auto-encoder method which enables nonlinear factorization of multivariate time series, learned end-to-end with a temporal deep learning latent space forecast model. By imposing a probabilistic latent space model, complex distributions of the input series are modeled via the decoder. Extensive experiments demonstrate that our model achieves state-of-the-art performance on many popular multivariate datasets, with gains sometimes as high as 50% for several standard metrics.

Adaptive Bayesian Spectral Analysis of High-Dimensional Nonstationary Time Series

This article introduces a nonparametric approach to spectral analysis of a high-dimensional multivariate nonstationary time series. The procedure is based on a novel frequency-domain factor model that provides a flexible yet parsimonious representation of spectral matrices from a large number of simultaneously observed time series. Real and imaginary parts of the factor loading matrices are modeled independently using a prior that is formulated from the tensor product of penalized splines and multiplicative gamma process shrinkage priors, allowing for infinitely many factors with loadings increasingly shrunk toward zero as the column index increases. Formulated in a fully Bayesian framework, the time series is adaptively partitioned into approximately stationary segments, where both the number and locations of partition points are assumed unknown. Stochastic approximation Monte Carlo techniques are used to accommodate the unknown number of segments, and a conditional Whittle likelihood-based Gibbs sampler is developed for efficient sampling within segments. By averaging over the distribution of partitions, the proposed method can approximate both abrupt and slowly varying changes in spectral matrices. Performance of the proposed model is evaluated by extensive simulations and demonstrated through the analysis of high-density electroencephalography. Supplementary materials for this article are available online.

A Bayesian State-Space Approach to Mapping Directional Brain Networks

The human brain is a directional network system of brain regions involving directional connectivity. Seizures are a directional network phenomenon as abnormal neuronal activities start from a seizure onset zone (SOZ) and propagate to otherwise healthy regions. To localize the SOZ of an epileptic patient, clinicians use intracranial electroencephalography (iEEG) to record the patient’s intracranial brain activity in many small regions. iEEG data are high-dimensional multivariate time series. We build a state-space multivariate autoregression (SSMAR) for iEEG data to model the underlying directional brain network. To produce scientifically interpretable network results, we incorporate into the SSMAR the scientific knowledge that the underlying brain network tends to have a cluster structure. Specifically, we assign to the SSMAR parameters a stochastic-blockmodel-motivated prior, which reflects the cluster structure. We develop a Bayesian framework to estimate the SSMAR, infer directional connections, and identify clusters for the unobserved network edges. The new method is robust to violations of model assumptions and outperforms existing network methods. By applying the new method to an epileptic patient’s iEEG data, we reveal seizure initiation and propagation in the patient’s directional brain network and discover a unique directional connectivity property of the SOZ. Overall, the network results obtained in this study bring new insights into epileptic patients’ normal and abnormal epileptic brain mechanisms and have the potential to assist neurologists and clinicians in localizing the SOZ—a long-standing research focus in epilepsy diagnosis and treatment. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

Multivariate time series prediction of high dimensional data based on deep reinforcement learning

In order to improve the prediction accuracy of high-dimensional data time series, a high-dimensional data multivariate time series prediction method based on deep reinforcement learning is proposed. The deep reinforcement learning method is used to solve the time delay of each variable and mine the data characteristics. According to the principle of maximum conditional entropy, the embedding dimension of the phase space is expanded, and a multivariate time series model of high-dimensional data is constructed. Thus, the conversion of reconstructed coordinates from low-dimensional to high-dimensional can be kept relatively stable. The strong independence and low redundancy of the final reconstructed phase space construct an effective model input vector for multivariate time series forecasting. Numerical experiments of classical multivariable chaotic time series show that the method proposed in this paper has better forecasting effect, which shows the forecasting effectiveness of this method.

The Convex Mixture Distribution: Granger Causality for Categorical Time Series.

We present a framework for learning Granger causality networks for multivariate categorical time series based on the mixture transition distribution (MTD) model. Traditionally, MTD is plagued by a nonconvex objective, non-identifiability, and presence of local optima. To circumvent these problems, we recast inference in the MTD as a convex problem. The new formulation facilitates the application of MTD to high-dimensional multivariate time series. As a baseline, we also formulate a multi-output logistic autoregressive model (mLTD), which while a straightforward extension of autoregressive Bernoulli generalized linear models, has not been previously applied to the analysis of multivariate categorial time series. We establish identifiability conditions of the MTD model and compare them to those for mLTD. We further devise novel and efficient optimization algorithms for MTD based on our proposed convex formulation, and compare the MTD and mLTD in both simulated and real data experiments. Finally, we establish consistency of the convex MTD in high dimensions. Our approach simultaneously provides a comparison of methods for network inference in categorical time series and opens the door to modern, regularized inference with the MTD model.

Granger causality detection in high-dimensional systems using feedforward neural networks

This paper proposes a novel methodology to detect Granger causality on average in vector autoregressive settings using feedforward neural networks. The approach accommodates unknown dependence structures between elements of high-dimensional multivariate time series with weak and strong persistence. To do this, we propose a two-stage procedure: first, we maximize the transfer of information between input and output variables in the network in order to obtain an optimal number of nodes in the intermediate hidden layers. Second, we apply a novel sparse double group lasso penalty function in order to identify the variables that have the predictive ability and, hence, indicate that Granger causality is present in the others. The penalty function inducing sparsity is applied to the weights that characterize the nodes of the neural network. We show the correct identification of these weights so as to increase sample sizes. We apply this method to the recently created Tobalaba network of renewable energy companies and show the increase in connectivity between companies after the creation of the network using Granger causality measures to map the connections.

Hybrid cycle reservoir with jumps for multivariate time series prediction: industrial application in oil drilling process

Industrial oil drilling processes usually produce high-dimensional multivariate time series data, in which the significant data changes associated with key variables possibly indicate potential drilling accidents. Therefore, it is of great significance to establish a multivariate time series prediction model for the safety of drilling operations. Recently, echo state networks (ESNs) have been widely used in multitime series predictions. However, traditional ESNs use randomly generated sparse network structures and single neuron models, which makes it difficult to achieve a satisfactory performance for complex multivariate time series predictions. In response to this problem, this study proposes a novel hybrid cycle reservoir with jumps (HCRJ) which combines the hybrid wavelet neuron model with the cycle reservoir with jumps (CRJ) structure. In the face of high-dimensional data sets generated by oil drilling, this paper selects a principal component analysis algorithm combined with certain process knowledge to find key variables before related variables are specified by means of Gray correlation analysis methods. The HCRJ networks are used to realize temporal predictions based on data of related variables. To confirm the validity of the model, the HCRJ networks are applied to an oil drilling process and compared to traditional ESNs and CRJ methods. The results show that the HCRJ networks enrich the dynamic characteristics of networks without increasing the complexity of the reserve pool, which helps improve the prediction accuracy of the oil drilling sequence and reduce the occurrence of drilling accidents.

Preface

Preface

Damage localization under ambient excitations and non-stationary vibration signals by a new hybrid algorithm for feature extraction and multivariate distance correlation methods

Ambient excitations applied to structures may lead to non-stationary vibration responses. In such circumstances, it may be difficult or improper to extract meaningful and significant damage features through methods that mainly rely on the stationarity of data. This article proposes a new hybrid algorithm for feature extraction as a combination of a new adaptive signal decomposition method called improved complete ensemble empirical mode decomposition with adaptive noise and autoregressive moving average model. The major contribution of this algorithm is to address the important issue of feature extraction under ambient vibration and non-stationary signals. The improved complete ensemble empirical mode decomposition with adaptive noise method is an improvement on the well-known ensemble empirical mode decomposition technique by removing redundant intrinsic mode functions. In addition, a novel automatic approach is presented to select the most relevant intrinsic mode functions to damage based on the intrinsic mode function energy level. Fitting an autoregressive moving average model to each selected intrinsic mode function, the model residuals are extracted as the damage-sensitive features. The main limitation is that such features are high-dimensional multivariate time series data, which may make a difficult and time-consuming decision-making process for damage localization. Multivariate distance correlation methods are introduced to cope with this drawback and locate structural damage using the multivariate residual sets of the normal and damaged conditions. The accuracy and robustness of the proposed methods are validated by a numerical shear-building model and an experimental benchmark structure. The effects of sampling frequency and time duration are evaluated as well. Results demonstrate the effectiveness and capability of the proposed methods to extract sufficient and reliable features, identify damage location, and quantify damage severity under ambient excitations and non-stationary signals.

High-Dimensional Multivariate Time Series With Additional Structure

ABSTRACTHigh-dimensional multivariate time series are challenging due to the dependent and high-dimensional nature of the data, but in many applications there is additional structure that can be exploited to reduce computing time along with statistical error. We consider high-dimensional vector autoregressive processes with spatial structure, a simple and common form of additional structure. We propose novel high-dimensional methods that take advantage of such structure without making model assumptions about how distance affects dependence. We provide nonasymptotic bounds on the statistical error of parameter estimators in high-dimensional settings and show that the proposed approach reduces the statistical error. An application to air pollution in the USA demonstrates that the estimation approach reduces both computing time and prediction error and gives rise to results that are meaningful from a scientific point of view, in contrast to high-dimensional methods that ignore spatial structure. In practice, these high-dimensional methods can be used to decompose high-dimensional multivariate time series into lower-dimensional multivariate time series that can be studied by other methods in more depth. Supplementary materials for this article are available online.

Short-Term Electric Load Forecasting Using Echo State Networks and PCA Decomposition

In this paper, we approach the problem of forecasting a time series (TS) of an electrical load measured on the Azienda Comunale Energia e Ambiente (ACEA) power grid, the company managing the electricity distribution in Rome, Italy, with an echo state network (ESN) considering two different leading times of 10 min and 1 day. We use a standard approach for predicting the load in the next 10 min, while, for a forecast horizon of one day, we represent the data with a high-dimensional multi-variate TS, where the number of variables is equivalent to the quantity of measurements registered in a day. Through the orthogonal transformation returned by PCA decomposition, we reduce the dimensionality of the TS to a lower number $k$ of distinct variables; this allows us to cast the original prediction problem in $k$ different one-step ahead predictions. The overall forecast can be effectively managed by $k$ distinct prediction models, whose outputs are combined together to obtain the final result. We employ a genetic algorithm for tuning the parameters of the ESN and compare its prediction accuracy with a standard autoregressive integrated moving average model.