Multivariate Time Series Prediction Research Articles

Enhanced Multi-variate Time Series Prediction Through Statistical-Deep Learning Integration: The VAR-Stacked LSTM Model

Enhanced Multi-variate Time Series Prediction Through Statistical-Deep Learning Integration: The VAR-Stacked LSTM Model

Integrating gated recurrent unit in graph neural network to improve infectious disease prediction: an attempt

ObjectiveThis study focuses on enhancing the precision of epidemic time series data prediction by integrating Gated Recurrent Unit (GRU) into a Graph Neural Network (GNN), forming the GRGNN. The accuracy of the GNN (Graph Neural Network) network with introduced GRU (Gated Recurrent Units) is validated by comparing it with seven commonly used prediction methods.MethodThe GRGNN methodology involves multivariate time series prediction using a GNN (Graph Neural Network) network improved by the integration of GRU (Gated Recurrent Units). Additionally, Graphical Fourier Transform (GFT) and Discrete Fourier Transform (DFT) are introduced. GFT captures inter-sequence correlations in the spectral domain, while DFT transforms data from the time domain to the frequency domain, revealing temporal node correlations. Following GFT and DFT, outbreak data are predicted through one-dimensional convolution and gated linear regression in the frequency domain, graph convolution in the spectral domain, and GRU (Gated Recurrent Units) in the time domain. The inverse transformation of GFT and DFT is employed, and final predictions are obtained after passing through a fully connected layer. Evaluation is conducted on three datasets: the COVID-19 datasets of 38 African countries and 42 European countries from worldometers, and the chickenpox dataset of 20 Hungarian regions from Kaggle. Metrics include Average Root Mean Square Error (ARMSE) and Average Mean Absolute Error (AMAE).ResultFor African COVID-19 dataset and Hungarian Chickenpox dataset, GRGNN consistently outperforms other methods in ARMSE and AMAE across various prediction step lengths. Optimal results are achieved even at extended prediction steps, highlighting the model’s robustness.ConclusionGRGNN proves effective in predicting epidemic time series data with high accuracy, demonstrating its potential in epidemic surveillance and early warning applications. However, further discussions and studies are warranted to refine its application and judgment methods, emphasizing the ongoing need for exploration and research in this domain.

A divide-and-rule combined learning method for truly multivariate time series prediction

A divide-and-rule combined learning method for truly multivariate time series prediction

DSTN: Dynamic Spatio-Temporal Network for Early Fault Warning in Chemical Processes

Multivariate time series prediction, especially in early fault warning for chemical processes, poses significant challenges. The advent of graph neural network (GNN) method has made breakthroughs in this domain by enabling the processing of topological data. However, the traditional methods suffer from the issue of over-smoothing and inability to capture intricate multi-scale spatio-temporal dependencies. Additionally, the existing graph structures fall short in describing the complex spatial relationships among multi-stage sensors, impeding their adaptability to dynamically evolving chain reaction scenarios. To alleviate these limitations, a novel Dynamic spatio-Temporal Network for early fault warning in chemical processes, named DSTN for short, is proposed in this paper. We extract the spatial and temporal features of the time series by the designed dynamic GNN and the improved Transformer network. Then, we integrate the spatio-temporal features through the residual network. DSTN has the following advantages: (1) A one-dimensional convolutional neural network is seamlessly incorporated into the Transformer architecture for bolstering its capacity to discern both global and local features within time series. (2) The continuous sliding window and mutual information methods are employed to construct a dynamic topology graph, and a K-order adjacency matrix is designed to rectify the inefficiencies in learning weights associated with convolution kernel parameters. (3) Multiple spatio-temporal modules interconnected via residual connection to adaptively fuse multi-scale features. Experimental results demonstrate that our proposed DSTN method outperforms existing methods in terms of both performance and interpretability in early fault warning of chemical processes.

A New Auto-Regressive Multi-Variable Modified Auto-Encoder for Multivariate Time-Series Prediction: A Case Study with Application to COVID-19 Pandemics.

The SARS-CoV-2 global pandemic prompted governments, institutions, and researchers to investigate its impact, developing strategies based on general indicators to make the most precise predictions possible. Approaches based on epidemiological models were used but the outcomes demonstrated forecasting with uncertainty due to insufficient or missing data. Besides the lack of data, machine-learning models including random forest, support vector regression, LSTM, Auto-encoders, and traditional time-series models such as Prophet and ARIMA were employed in the task, achieving remarkable results with limited effectiveness. Some of these methodologies have precision constraints in dealing with multi-variable inputs, which are important for problems like pandemics that require short and long-term forecasting. Given the under-supply in this scenario, we propose a novel approach for time-series prediction based on stacking auto-encoder structures using three variations of the same model for the training step and weight adjustment to evaluate its forecasting performance. We conducted comparison experiments with previously published data on COVID-19 cases, deaths, temperature, humidity, and air quality index (AQI) in São Paulo City, Brazil. Additionally, we used the percentage of COVID-19 cases from the top ten affected countries worldwide until May 4th, 2020. The results show 80.7% and 10.3% decrease in RMSE to entire and test data over the distribution of 50 trial-trained models, respectively, compared to the first experiment comparison. Also, model type#3 achieved 4th better overall ranking performance, overcoming the NBEATS, Prophet, and Glounts time-series models in the second experiment comparison. This model shows promising forecast capacity and versatility across different input dataset lengths, making it a prominent forecasting model for time-series tasks.

Is Single Enough? A Joint Spatiotemporal Feature Learning Framework for Multivariate Time Series Prediction.

A fuzzy cognitive map (FCM) is a simple but effective tool for modeling and predicting time series. This article focuses on the problem of multivariate time series prediction (TSP), which is essential and challenging in data mining. Although several FCM-based approaches have been designed to solve this problem, their feature extraction module designed for single mode falls short in capturing the nonlinear spatiotemporal dependencies among variates, thereby resulting in low prediction accuracy in forecasting multivariate time series, which shows that the single mode learning is not enough. Therefore, in this article, we propose a joint spatiotemporal feature learning framework for multivariate TSP, where a mix-resolution spatial module consisting of multiple sparse autoencoders (SAEs) is designed to extract the feature series with different spatial resolutions, and a mix-order spatiotemporal module concluding multiple high-order FCMs (HFCMs) is designed to model the spatiotemporal dynamics of these feature series. Finally, the outputs of the two modules are concatenated to predict future values. We refer to this framework as the spatiotemporal FCM (STFCM). Especially, an efficient learning algorithm is designed to update the integral weights of STFCM based on the batch gradient descent algorithm when it deems necessary. We validate the performance of the STFCM on four real-world datasets. Compared with the existing state-of-the-art (SOTA) methods, the experimental results not only show the advantages of the two designed modules in the STFCM but also show the excellent performance of the STFCM.

Latent Diffusion Transformer for Probabilistic Time Series Forecasting

The probability prediction of multivariate time series is a notoriously challenging but practical task. This research proposes to condense high-dimensional multivariate time series forecasting into a problem of latent space time series generation, to improve the expressiveness of each timestamp and make forecasting more manageable. To solve the problem that the existing work is hard to extend to high-dimensional multivariate time series, we present a latent multivariate time series diffusion framework called Latent Diffusion Transformer (LDT), which consists of a symmetric statistics-aware autoencoder and a diffusion-based conditional generator, to implement this idea. Through careful design, the time series autoencoder can compress multivariate timestamp patterns into a concise latent representation by considering dynamic statistics. Then, the diffusion-based conditional generator is able to efficiently generate realistic multivariate timestamp values on a continuous latent space under a novel self-conditioning guidance which is modeled in a non-autoregressive way. Extensive experiments demonstrate that our model achieves state-of-the-art performance on many popular high-dimensional multivariate time series datasets.

HDMixer: Hierarchical Dependency with Extendable Patch for Multivariate Time Series Forecasting

Multivariate time series (MTS) prediction has been widely adopted in various scenarios. Recently, some methods have employed patching to enhance local semantics and improve model performance. However, length-fixed patch are prone to losing temporal boundary information, such as complete peaks and periods. Moreover, existing methods mainly focus on modeling long-term dependencies across patches, while paying little attention to other dimensions (e.g., short-term dependencies within patches and complex interactions among cross-variavle patches). To address these challenges, we propose a pure MLP-based HDMixer, aiming to acquire patches with richer semantic information and efficiently modeling hierarchical interactions. Specifically, we design a Length-Extendable Patcher (LEP) tailored to MTS, which enriches the boundary information of patches and alleviates semantic incoherence in series. Subsequently, we devise a Hierarchical Dependency Explorer (HDE) based on pure MLPs. This explorer effectively models short-term dependencies within patches, long-term dependencies across patches, and complex interactions among variables. Extensive experiments on 9 real-world datasets demonstrate the superiority of our approach. The code is available at https://github.com/hqh0728/HDMixer.

CTF-former: A novel simplified multi-task learning strategy for simultaneous multivariate chaotic time series prediction

Multivariate chaotic time series prediction is a challenging task, especially when multiple variables are predicted simultaneously. For multiple related prediction tasks typically require multiple models, however, multiple models are difficult to keep synchronization, making immediate communication between predicted values challenging. Although multi-task learning can be applied to this problem, the principles of allocation and layout options between shared and specific representations are ambiguous. To address this issue, a novel simplified multi-task learning method was proposed for the precise implementation of simultaneous multiple chaotic time series prediction tasks. The scheme proposed consists of a cross-convolution operator designed to capture variable correlations and sequence correlations, and an attention module proposed to capture the information embedded in the sequence structure. In the attention module, a non-linear transformation was implemented with convolution, and its local receptive field and the global dependency of the attention mechanism achieve complementarity. In addition, an attention weight calculation was devised that takes into account not only the synergy of time and frequency domain features, but also the fusion of series and channel information. Notably the scheme proposed a purely simplified design principle of multi-task learning by reducing the specific network to single neuron. The precision of the proposed solution and its potential for engineering applications were verified with the Lorenz system and power consumption. The mean absolute error of the proposed method was reduced by an average of 82.9% in the Lorenz system and 19.83% in power consumption compared to the Gated Recurrent Unit.

GRAformer: A gated residual attention transformer for multivariate time series forecasting

Recurrent Neural Networks (RNNs), particularly when equipped with output windows – a standard practice in contemporary time series forecasting – have shown proficiency in handling short-term dependencies. Nonetheless, RNNs can encounter challenges in maintaining hidden states over extended forecasting periods, particularly in longer-term predictions where increased hidden state sizes and extended look-back windows can lead to gradient instability. In contrast, Transformer-based models, with their distinctive architecture designed to encode complex contextual relationships and enable computations to be done in parallel, are emerging as a popular alternative in this field. However, current research has mainly focused on modifying attention mechanisms, overlooking opportunities to improve the feedforward layer, which could lead to efficiency limitations. Moreover, prevailing methods often assume absolute independence between channels, disregarding distinct features among variables and failing to fully leverage channel-specific information. To address these gaps, we propose an efficient transformer design for multivariate time series prediction. Our approach integrates two key components: (i) a gated residual attention unit that enhances predictive accuracy and computational efficiency, and (ii) a channel embedding technique that differentiates between series and boosts performance. Theoretically, we prove that our model has recurrent dynamics introduced by the RNN layer. Through extensive experiments on real-world data, we demonstrate that our proposed method achieves competitive predictive accuracy compared to prior approaches while exhibiting accelerated processing relative to state-of-the-art transformers. Our code, data, and trained models are available at https://github.com/MythosAd/GRAformer.

An enhanced abnormal information expression spatiotemporal model for anomaly detection in multivariate time-series

The detection of anomalies in high-dimensional time-series has always played a crucial role in the domain of system security. Recently, with rapid advancements in transformer model and graph neural network (GNN) technologies, spatiotemporal modeling approaches for anomaly detection tasks have been greatly improved. However, most methods focus on optimizing upstream time-series prediction tasks by leveraging joint spatiotemporal features. Through experiments, we found that this modeling approach not only risks the loss of some original anomaly information during data preprocessing, but also focuses on optimizing the performance of the upstream prediction task and does not directly enhance the performance of the downstream detection task. We propose a spatiotemporal anomaly detection model that incorporates an improved attention mechanism in the process of temporal modeling. We adopt a heterogeneous graph contrastive learning approach in spatio modeling to compensate for the representation of anomalous behavioral information, thereby guiding the model through thorough training. Through validation on two widely used real-world datasets, we demonstrate that our model outperforms baseline methods. We also explore the impact of multivariate time-series prediction tasks on the detection task, and visualize the reasons behind the benefits gained by our model.

A fuzzy C-means clustering-based hybrid multivariate time series prediction framework with feature selection

A fuzzy C-means clustering-based hybrid multivariate time series prediction framework with feature selection

MR-Transformer: Multiresolution Transformer for Multivariate Time Series Prediction.

Multivariate time series (MTS) prediction has been studied broadly, which is widely applied in real-world applications. Recently, transformer-based methods have shown the potential in this task for their strong sequence modeling ability. Despite progress, these methods pay little attention to extracting short-term information in the context, while short-term patterns play an essential role in reflecting local temporal dynamics. Moreover, we argue that there are both consistent and specific characteristics among multiple variables, which should be fully considered for MTS modeling. To this end, we propose a multiresolution transformer (MR-Transformer) for MTS prediction, modeling MTS from both the temporal and the variable resolution. Specifically, for the temporal resolution, we design a long short-term transformer. We first split the sequence into nonoverlapping segments in an adaptive way and then extract short-term patterns within segments, while long-term patterns are captured by the inherent attention mechanism. Both of them are aggregated together to capture the temporal dependencies. For the variable resolution, besides the variable-consistent features learned by long short-term transformer, we also design a temporal convolution module to capture the specific features of each variable individually. MR-Transformer enhances the MTS modeling ability by combining multiresolution features between both time steps and variables. Extensive experiments conducted on real-world time series datasets show that MR-Transformer significantly outperforms the state-of-the-art MTS prediction models. The visualization analysis also demonstrates the effectiveness of the proposed model.

Dynamic multi-fusion spatio-temporal graph neural network for multivariate time series forecasting

Multivariate time series data with multiple points is a kind of spatio-temporal data, and in recent years, spatio-temporal graph neural networks (STGNNs) have become one of the most effective methods for spatio-temporal data prediction. By leveraging graph neural networks and sequence models, STGNNs simultaneously capture the hidden temporal and spatial patterns in spatio-temporal data. However, there exist two key limitations: (1) Most existing STGNNs whether based on static graphs or dynamic graphs, fail to introduce time periodicity into graph structure learning and ignore the importance of multi-relational graph structure learning. (2) Most multivariate time series prediction models with multiple points ignore the relation between multiple attributes. To this end, we propose a novel framework called the Dynamic Multi-fusion Spatio-temporal Graph Neural Network (DMF-STNet), where multi-fusion embodies two meanings: spatio-temporal fusion and multi-attribute fusion, as well as multi-graph fusion and multi-layer fusion. The dynamic periodic graph is proposed to mine time periodicity that is hidden in graph structures, and we further offer multi-relational graph structure learning methods to capture static and dynamic relations. Moreover, we introduce a dual-stage feature aggregation method to aggregate hidden features from both primary and auxiliary attributes. Extensive experiments on three real-world datasets demonstrate the efficacy of our proposed DMF-STNet, and the visualization study proves the interpretability of learned weights.

Predicting scoring trends in basketball games based on multivariate time series analysis

Abstract In this paper, we first study the evaluation framework of Granger causality between time series. Then, the multivariate time series prediction model is constructed based on transfer entropy and graph attention network. The multivariate time series prediction model architecture has causal branches with transfer entropy, and the prediction module is executed using graph attention network. Finally, the accuracy of basketball game score prediction is analyzed, as well as comparing the accuracy of the multivariate time series prediction model with other models. The number of offensive rounds in basketball games can be predicted with a high accuracy of 67.94% using the multivariate time series prediction model. For the prediction of the scoring trend of player rotation, the ABHJI combination rotating on the court is the most efficient in scoring, and the average value of scoring is as high as 85.463. The average accuracy of the TSP reaches 72.2%, and the model has high feasibility and is suitable for scoring prediction in basketball games.

LSTM time series NDVI prediction method incorporating climate elements: A case study of Yellow River Basin, China

Accurate prediction of the trend of Normalized Difference Vegetation Index (NDVI) time series in the Yellow River Basin (YRB) is crucial for the assessment of the hydrological and ecological environment in this region. Currently, the NDVI time series prediction model is primarily based on traditional models and single-variable neural network models. Nevertheless, these models present challenges in considering the limitations of multiple factors, causing the NDVI time series prediction results to lack reliability. To predict NDVI time-series in the YRB of China, this study constructed a multilayer multivariate Long-Short Term Memory (LSTM) neural network model including climatic components. The initial important climatic elements in this region were identified using GeoDetector. Then, the relationship between NDVI and climatic factors in the YRB of China is established. Finally, numerical scale data are used to train and predict a multilayer multivariate LSTM model with climatic components. According to the results, the three-layer multivariate LSTM neural network NDVI time series prediction model developed in this study has the best performance among the evaluated indices. When compared to existing time series prediction models, the proposed model in this study takes into account the common constraint effect of various climate factors on NDVI. This leads to a significantly improved prediction accuracy, presenting new opportunities for enhancing the prediction model. By analyzing the NDVI time series prediction outcomes for the YRB, it has been determined that the ecological environment of the area will continuously improve in the future. This study offers significant technological and theoretical backing for assessing the hydrological and ecological environment of the YRB and comparable ecologically vulnerable regions in China.

Multivariate time series prediction for CO2 concentration and flowrate of flue gas from biomass-fired power plants

Integrating CO2 capture with biomass-fired combined heat and power (bio-CHP) plants is a promising method to achieve negative emissions. However, the use of versatile biomass, including waste, and the dynamic operation of bio-CHP plants leads to large fluctuations in the flowrate and CO2 concentration of the flue gas (FG), which further affect the operation of post-combustion CO2 capture. To optimize the dynamic operation of CO2 capture, a reliable model to predict the FG flowrate and CO2 concentration in real time is essential. In this paper, a data-driven model based on the Transformer architecture is developed. The model validation shows that the root mean squared error (RMSE), mean absolute percentage error (MAPE), and Pearson correlation coefficient (PPMCC) of Transformer are 0.3553, 0.0189, and 0.8099 respectively for the prediction of FG flowrate; and 13.137, 0.0318, and 0.8336 respectively for the prediction of CO2 concentration. The potential impact of various meteorological parameters on model accuracy is also assessed by analyzing the Shapley value. It is found that temperature and direct horizontal irradiance (DHI) are the most important factors, which should be selected as input features. In addition, using the near-infrared (NIR) spectral data as input features is also found to be an effective way to improve the prediction accuracy. It can reduce RMSE and MAPE for CO2 concentration from 0.2982 to 0.2887 and 0.0158 to 0.0157 respectively, and RMSE and MAPE for FG flowrate from 4.9854 to 4.7537 and 0.0141 to 0.0121 respectively. The Transformer model is also compared to other models, including long short-term memory network (LSTM) and artificial neural network (ANN), which results show that the Transformer model is superior in predicting complex dynamic patterns and nonlinear relationships.

Mixformer: An improved self-attention architecture applied to multivariate chaotic time series prediction

Multivariate chaotic time series prediction has been a challenging problem, especially the coupling relationship between multiple variables need to be carefully considered. We propose a self-attention architecture with an information interaction module, called mixformer, applied to the multivariate chaotic time series prediction task. First, Based on rethinking the multivariate data structure, we compensate the deficiency of phase space reconstruction with a group of cross-convolution operator that can automatically update parameters, and propose an explicitly designed feature reconstruction module. Then, to address the problem of interactive fusion of series information and channel information, we propose an information interaction module that enables the feature communication by expanding and contracting dimensions. By breaking the communication barrier between series information and channel information, the feature representation capability is enhanced. Finally, we construct mixformer that combines locally sparse features and global context features. And notably, mixformer integrates the information interaction module and the feature reconstruction module to form a continuous solution. By comparing the results with existing models using multiple simulated data (Lorenz, Chen, and Rossler systems) and real-world application (power consumption), it is verified that our proposed model achieves surprising performance in practice.

A WSFA-based adaptive feature extraction method for multivariate time series prediction

A WSFA-based adaptive feature extraction method for multivariate time series prediction

Multivariate multi-step time series prediction of induction motor situation based on fused temporal-spatial features

Induction motor temperature situation prediction provides a decision basis for preventive maintenance in coal mining companies. However, multi-step prediction of induction motor temperature is a challenge due to the complexity of working conditions and external disturbances in surface coal mines. This paper proposes a multi-sensor fusion multi-step prediction model based on Graph Convolutional Neural Network with Long Short-Term Memory Network (GCN-LSTM). Specifically, the model takes into account the spatial correlation and long-term temporal dependence of multi-source sensors as well as the temporal-spatial fusion correlation at different times. This thesis is based on multi-source temperature sequence data collected from a mining induction motor. Experimental results show that the model is able to achieve 31.3%, 38.7%, and 17.1% performance improvement compared to CNN, LSTM, and GCN methods.