Multivariate Time Series Forecasting Research Articles

VTNet: A multi-domain information fusion model for long-term multi-variate time series forecasting with application in irrigation water level

Time series forecasting is intricately tied to production and life, garnering widespread attention over an extended period. Enhancing the performance of long-term multivariate time series forecasting (MTSF) poses a highly challenging task, as it requires mining complicated and obscure temporal patterns in many aspects. For this reason, this paper proposes a long-term forecasting model based on multi-domain fusion (VTNet) to adaptively capture and refine multi-scale intra- and inter-variate dependencies. In contrast to previous techniques, we devise a dual-stream learning architecture. Firstly, the fast Fourier Transform (FFT) is adopted to extract frequency domain information. The original sequences are then transformed into 2D visual features in the temporal-frequency domain, and a 2D-TBlock is designed for multi-scale dynamic learning. Secondly, a combination of convolution and recurrent networks continues to explore the local temporal features and preserve the global trend. Finally, multi-modal circulant fusion is applied to achieve a more comprehensive and enriched feature fusion representation, further promoting overall performance. Extensive experiments are conducted on 9 public benchmark datasets and the real-world irrigation water level to showcase VTNet’s promoted performance and generalization. Moreover, VTNet yields 46.93% and 25.36% relative improvements for water level forecasting, revealing its potential application value in water-saving planning and extreme event early warning.

Leveraging Hybrid Deep Learning Models for Enhanced Multivariate Time Series Forecasting

Time series forecasting is crucial in various domains, ranging from finance and economics to weather prediction and supply chain management. Traditional statistical methods and machine learning models have been widely used for this task. However, they often face limitations in capturing complex temporal dependencies and handling multivariate time series data. In recent years, deep learning models have emerged as a promising solution for overcoming these limitations. This paper investigates how deep learning, specifically hybrid models, can enhance time series forecasting and address the shortcomings of traditional approaches. This dual capability handles intricate variable interdependencies and non-stationarities in multivariate forecasting. Our results show that the hybrid models achieved lower error rates and higher R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$R^2$$\\end{document} values, signifying their superior predictive performance and generalization capabilities. These architectures effectively extract spatial features and temporal dynamics in multivariate time series by combining convolutional and recurrent modules. This study evaluates deep learning models, specifically hybrid architectures, for multivariate time series forecasting. On two real-world datasets - Traffic Volume and Air Quality - the TCN-BiLSTM model achieved the best overall performance. For Traffic Volume, the TCN-BiLSTM model achieved an R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$R^2$$\\end{document} score of 0.976, and for Air Quality, it reached an R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$R^2$$\\end{document} score of 0.94. These results highlight the model’s effectiveness in leveraging the strengths of Temporal Convolutional Networks (TCNs) for capturing multi-scale temporal patterns and Bidirectional Long Short-Term Memory (BiLSTMs) for retaining contextual information, thereby enhancing the accuracy of time series forecasting.

XLSTMTime: Long-Term Time Series Forecasting with xLSTM

In recent years, transformer-based models have gained prominence in multivariate long-term time series forecasting (LTSF), demonstrating significant advancements despite facing challenges such as high computational demands, difficulty in capturing temporal dynamics, and managing long-term dependencies. The emergence of LTSF-Linear, with its straightforward linear architecture, has notably outperformed transformer-based counterparts, prompting a reevaluation of the transformer’s utility in time series forecasting. In response, this paper presents an adaptation of a recent architecture, termed extended LSTM (xLSTM), for LTSF. xLSTM incorporates exponential gating and a revised memory structure with higher capacity that has good potential for LTSF. Our adopted architecture for LTSF, termed xLSTMTime, surpasses current approaches. We compare xLSTMTime’s performance against various state-of-the-art models across multiple real-world datasets, demonstrating superior forecasting capabilities. Our findings suggest that refined recurrent architectures can offer competitive alternatives to transformer-based models in LTSF tasks, potentially redefining the landscape of time series forecasting.

FTMLP: MLP with Feature-Temporal Block for multivariate time series forecasting

Multivariate time series (MTS) forecasting is extensively applied in real-world scenarios. Recent studies introduced the concept of channel independence, which has achieved significant results. However, this approach often misses the intricate interrelations among features and lacks physical interpretation. To address this problem, we propose a new framework, the Feature-Temporal block, designed to extract both temporal and feature information. Each block contains two modules: a feature module using a gating mechanism to understand competitive interactions among features, and a temporal module using learnable filters for frequency domain filtering. This innovative design allows FTMLP to combine information across feature and sequence dimensions, as well as time and frequency domains. The computational efficiency of FTMLP is enhanced due to its exclusive use of Multi-Layer Perceptron (MLP). Comprehensive experiments on several real-world datasets demonstrate that FTMLP achieves state-of-the-art results in terms of overall performance. Code is available at https://github.com/whxlearning/FTMLP.

Multi-attention network with redundant information filtering for multi-horizon forecasting in multivariate time series

Multi-horizon forecasting of multivariate time series has always been a prominent research topic in domains such as finance and transportation. While prediction models that integrate attention mechanisms and Seq2Seq frameworks effectively capture the temporal dependencies between sequences, there might exist distinctions in the contributions of different input variables to the predictions. RNN-based encoders often overlook these distinctions. Hence, we propose a Multi-attention Network with Redundant Information Filtering (MNRIF) for Multi-horizon Forecasting in Multivariate Time Series, comprising the underlying component Two-stage Feature Fusion Network (TSFFN) and the upper-level structure Temporal Attention Network(TAN). TSFFN first designs a variable-level attention to screen the spatial semantics of explanatory variables during multi-step recursive iterations, effectively reducing noisy information, then builds a feature gate fusion module to adaptively weigh the degree of association between prediction targets and spatial semantics as well as target variables, thereby accomplishing secondary filtering of redundant information. TAN is a task-oriented learner that combines the Encoder–Decoder framework with attention mechanism. By means of dual information filtering based on attention mechanism and recurrent gate, it enhances effective mining of long-term temporal dependency features in sequences. Extensive experiments conducted on three real-world benchmark datasets demonstrate MNRIF’s superior performance compared to various state-of-the-art methods of different types.

Constructing Spatial Relationship and Temporal Relationship Oriented Composite Fuzzy Cognitive Maps for Multivariate Time Series Forecasting

Constructing Spatial Relationship and Temporal Relationship Oriented Composite Fuzzy Cognitive Maps for Multivariate Time Series Forecasting

Enhancing Multivariate Time Series Forecasting: A Novel Approach with Mallows Model Averaging and Graph Neural Networks

Enhancing Multivariate Time Series Forecasting: A Novel Approach with Mallows Model Averaging and Graph Neural Networks

A dual-topological graph memory network for anti-noise multivariate time series forecasting

Multivariate time series (MTS) forecasting plays an essential role in the automation and optimization process of intelligent applications. However, capturing correlations and dependencies among variables in MTS data remains a major challenge for the forecasting models. Although existing methods explain these complex relationships by deeply analyzing variables and intervariable dependencies in MTS data, they tend to be affected by the noise interference prevalent in real-world data, which increases the difficulty for forecasting models to characterize the real features, and thus makes it difficult to achieve satisfactory forecasting results. To address these challenges, this paper designs a dual-topological graph memory (DualGM) network for anti-noise multivariate time series forecasting. The core idea is to analyze the complex relationships in MTS data in detail by constructing dual-topological graphs and to learn and retain the comprehensive information of sequence features by employing a bidirectional self-attention memory module. Specifically, the model first constructs the MTS data as a graph structure and creates a probability topological graph and a distance topological graph based on the similarity between the samples, which helps the model recognize and filter the noise more efficiently in noisy environments, and valuable information is extracted from the noisy data. Then, the topological neighborhood features of the MTS data in each layer are summarized via multilayer graph groups, and the probability and distance spatial information is fused using a bidirectional self-attention mechanism to further enhance the memory and retention of the topological features, thus improving the accuracy of the forecasting model. The experimental results on four real datasets demonstrate that the DualGM model outperforms previously reported methods under various settings; in particular, the RSE values of the DualGM model are higher than those of the second-best results by 12%, 37%, 0.8% and 16% on the four datasets. Moreover, the DualGM model provides a valuable solution for multivariate time series forecasting under noise interference.

Variational autoencoders and transformers for multivariate time-series generative modeling and forecasting: Applications to vortex-induced vibrations

This study employs a data-driven approach to studying physical system vibrations, focusing on two main aspects: using variational autoencoders (VAEs) to generate physical data (i.e. data “similar” to those obtained via real-world processes) and using transformers in order to continuously forecast flexible body nonstationary vibrations (2D time-series) in time–space using information from sparse sensors on the body (observers). A VAE is trained on vortex-induced vibrations (VIV) data collected from experiments conducted by the authors and is then tasked with generating synthetic VIV data similar to the experimental. The synthetic data are then used to train a transformer architecture whose objective is to continuously forecast the vibrations in time–space using sparse observations. The transformer (which has never seen real data) is tested against real experiments and its performance is compared to that of the same architecture trained on real data. In doing so, the ability of VAEs to generate data which preserve their training data’s intrinsic properties (i.e. physicality) is evaluated. Finally a comparison between the forecasting performance of the transformer architecture, an LSTM, and a DNN is presented.

Sparse transformer with local and seasonal adaptation for multivariate time series forecasting

Transformers have achieved remarkable performance in multivariate time series(MTS) forecasting due to their capability to capture long-term dependencies. However, the canonical attention mechanism has two key limitations: (1) its quadratic time complexity limits the sequence length, and (2) it generates future values from the entire historical sequence. To address this, we propose a Dozer Attention mechanism consisting of three sparse components: (1) Local, each query exclusively attends to keys within a localized window of neighboring time steps. (2) Stride, enables each query to attend to keys at predefined intervals. (3) Vary, allows queries to selectively attend to keys from a subset of the historical sequence. Notably, the size of this subset dynamically expands as forecasting horizons extend. Those three components are designed to capture essential attributes of MTS data, including locality, seasonality, and global temporal dependencies. Additionally, we present the Dozerformer Framework, incorporating the Dozer Attention mechanism for the MTS forecasting task. We evaluated the proposed Dozerformer framework with recent state-of-the-art methods on nine benchmark datasets and confirmed its superior performance. The experimental results indicate that excluding a subset of historical time steps from the time series forecasting process does not compromise accuracy while significantly improving efficiency. Code is available at https://github.com/GRYGY1215/Dozerformer.

Multiview Spatial-Temporal Meta-Learning for Multivariate Time Series Forecasting.

Multivariate time series modeling has been essential in sensor-based data mining tasks. However, capturing complex dynamics caused by intra-variable (temporal) and inter-variable (spatial) relationships while simultaneously taking into account evolving data distributions is a non-trivial task, which faces accumulated computational overhead and multiple temporal patterns or distribution modes. Most existing methods focus on the former direction without adaptive task-specific learning ability. To this end, we developed a holistic spatial-temporal meta-learning probabilistic inference framework, entitled ST-MeLaPI, for the efficient and versatile learning of complex dynamics. Specifically, first, a multivariate relationship recognition module is utilized to learn task-specific inter-variable dependencies. Then, a multiview meta-learning and probabilistic inference strategy was designed to learn shared parameters while enabling the fast and flexible learning of task-specific parameters for different batches. At the core are spatial dependency-oriented and temporal pattern-oriented meta-learning approximate probabilistic inference modules, which can quickly adapt to changing environments via stochastic neurons at each timestamp. Finally, a gated aggregation scheme is leveraged to realize appropriate information selection for the generative style prediction. We benchmarked our approach against state-of-the-art methods with real-world data. The experimental results demonstrate the superiority of our approach over the baselines.

Dynamic Spatial-Temporal Embedding via Neural Conditional Random Field for Multivariate Time Series Forecasting

Dynamic Spatial-Temporal Embedding via Neural Conditional Random Field for Multivariate Time Series Forecasting

Frequency-Enhanced Transformer with Symmetry-Based Lightweight Multi-Representation for Multivariate Time Series Forecasting

Frequency-Enhanced Transformer with Symmetry-Based Lightweight Multi-Representation for Multivariate Time Series Forecasting

Functional Relation Field: A Model-Agnostic Framework for Multivariate Time Series Forecasting

In multivariate time series forecasting, the most popular strategy for modeling the relationship between multiple time series is the construction of graph, where each time series is represented as a node and related nodes are connected by edges. However, the relationship between multiple time series is typically complicated, e.g. the sum of outflows from upstream nodes may be equal to the inflows of downstream nodes. Such relations widely exist in many real-world scenarios for multivariate time series forecasting, yet are far from well studied. In these cases, graph might be insufficient for modeling the complex dependency between nodes. To this end, we explore a new framework to model the inter-node relationship in a more precise way based our proposed inductive bias, Functional Relation Field, where a group of functions parameterized by neural networks are learned to characterize the dependency between multiple time series. Essentially, these learned functions then form a “field”, i.e. a particular set of constraints, to regularize the training loss of the backbone prediction network and enforce the inference process to satisfy these constraints. Since our framework introduces the relationship bias in a data-driven manner, it is flexible and model-agnostic such that it can be applied to any existing multivariate time series prediction networks for boosting performance. The experiment is conducted on one toy dataset to show our approach can well recover the true constraint relationship between nodes. And various real-world datasets are also considered with different backbone prediction networks. Results show that the prediction error can be reduced remarkably with the aid of the proposed framework.

TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods

Time series are generated in diverse domains such as economic, traffic, health, and energy, where forecasting of future values has numerous important applications. Not surprisingly, many forecasting methods are being proposed. To ensure progress, it is essential to be able to study and compare such methods empirically in a comprehensive and reliable manner. To achieve this, we propose TFB, an automated benchmark for Time Series Forecasting (TSF) methods. TFB advances the state-of-the-art by addressing shortcomings related to datasets, comparison methods, and evaluation pipelines: 1) insufficient coverage of data domains, 2) stereotype bias against traditional methods, and 3) inconsistent and inflexible pipelines. To achieve better domain coverage, we include datasets from 10 different domains : traffic, electricity, energy, the environment, nature, economic, stock markets, banking, health, and the web. We also provide a time series characterization to ensure that the selected datasets are comprehensive. To remove biases against some methods, we include a diverse range of methods, including statistical learning, machine learning, and deep learning methods, and we also support a variety of evaluation strategies and metrics to ensure a more comprehensive evaluations of different methods. To support the integration of different methods into the benchmark and enable fair comparisons, TFB features a flexible and scalable pipeline that eliminates biases. Next, we employ TFB to perform a thorough evaluation of 21 Univariate Time Series Forecasting (UTSF) methods on 8,068 univariate time series and 14 Multivariate Time Series Forecasting (MTSF) methods on 25 datasets. The results offer a deeper understanding of the forecasting methods, allowing us to better select the ones that are most suitable for particular datasets and settings. Overall, TFB and this evaluation provide researchers with improved means of designing new TSF methods.

WITHDRAWN: FCDNet: Frequency-guided complementary dependency modeling for multivariate time-series forecasting

Multivariate time-series (MTS) forecasting is a challenging task in many real-world non-stationary dynamic scenarios. In addition to intra-series temporal signals, the inter-series dependency also plays a crucial role in shaping future trends. How to enable the model’s awareness of dependency information has raised substantial research attention. Previous approaches have either presupposed dependency constraints based on domain knowledge or imposed them using real-time feature similarity. However, MTS data often exhibit both enduring long-term stable relationships and transient short-term interactions, which mutually influence their evolving states. It is necessary to recognize and incorporate the complementary dependencies for more accurate MTS forecasting. The frequency information in time series reflects the evolutionary rules behind complex temporal dynamics, and different frequency components can be used to well construct interactive dependency structures with varying states between variables. To this end, we propose FCDNet, a concise yet effective complementary dependency modeling framework for multivariate time-series forecasting. Specifically, FCDNet overcomes the above limitations by applying two lightweight dependency constructors to help extract stable static and dynamic varying dependency information adaptively from long short-term multi-level frequency patterns. With the growth of input variables, the number of trainable parameters in FCDNet only increases linearly, which is conducive to the model’s scalability and avoids over-fitting. Additionally, adopting a frequency-based perspective can effectively mitigate the influence of noise within MTS data, which helps capture more genuine dependencies. The experimental results on six real-world datasets from multiple fields show that FCDNet significantly exceeds strong baselines, with an average improvement of 6.82% on MAE, 4.98% on RMSE, and 4.91% on MAPE. In addition, the ability of FCDNet to jointly learn high-quality static and dynamic graph structures is demonstrated empirically. Our source codes are publicly available at https://github.com/onceCWJ/FCDNet.

Deep Coupling Network for Multivariate Time Series Forecasting

Multivariate time series (MTS) forecasting is crucial in many real-world applications. To achieve accurate MTS forecasting, it is essential to simultaneously consider both intra- and inter-series relationships among time series data. However, previous work has typically modeled intra- and inter-series relationships separately and has disregarded multi-order interactions present within and between time series data, which can seriously degrade forecasting accuracy. In this article, we reexamine intra- and inter-series relationships from the perspective of mutual information and accordingly construct a comprehensive relationship learning mechanism tailored to simultaneously capture the intricate multi-order intra- and inter-series couplings. Based on the mechanism, we propose a novel deep coupling network for MTS forecasting, named DeepCN, which consists of a coupling mechanism dedicated to explicitly exploring the multi-order intra- and inter-series relationships among time series data concurrently, a coupled variable representation module aimed at encoding diverse variable patterns, and an inference module facilitating predictions through one forward step. Extensive experiments conducted on seven real-world datasets demonstrate that our proposed DeepCN achieves superior performance compared with the state-of-the-art baselines.

TimeSQL: Improving multivariate time series forecasting with multi-scale patching and smooth quadratic loss

Multivariate time series are usually sequences of real-valued variables recorded at regular intervals. Forecasting them poses significant challenges due to their inherent noise and complex temporal dynamics. To effectively address these challenges, we introduce TimeSQL, a novel approach specifically designed for multivariate time series. TimeSQL employs multi-scale patching and a smooth quadratic loss (SQL) function. This enables it to adeptly capture both local and long-term patterns in complex, noisy data environments. The multi-scale patching approach provides a comprehensive view of temporal correlations, while the SQL, rooted in the rational quadratic kernel, strategically adjusts gradients to avoid overfitting. Through both theoretical and empirical analysis, we have demonstrated that TimeSQL exhibits superior noise resilience compared with models based on Mean Squared Error (MSE). Evaluated across eight diverse benchmark datasets, TimeSQL consistently delivers exceptional performance in multivariate time series forecasting. Furthermore, ablation studies underscore the adaptability of TimeSQL's components, enhancing the performance of various other forecasting models and proving their utility as plug-and-play modules.

Electricity demand forecasting at distribution and household levels using explainable causal graph neural network

Forecasting electricity demand is an essential part of the smart grid to ensure a stable and reliable power grid. With the increasing integration of renewable energy resources into the grid, forecasting the demand for electricity is critical at all levels, from the distribution to the household. Most existing forecasting methods, however, can be considered black-box models as a result of deep digitalization enablers, such as deep neural networks, which remain difficult to interpret by humans. Moreover, capture of the inter-dependencies among variables presents a significant challenge for multivariate time series forecasting. In this paper we propose eXplainable Causal Graph Neural Network (X-CGNN) for multivariate electricity demand forecasting that overcomes these limitations. As part of this method, we have intrinsic and global explanations based on causal inferences as well as local explanations based on post-hoc analyses. We have performed extensive validation on two real-world electricity demand datasets from both the household and distribution levels to demonstrate that our proposed method achieves state-of-the-art performance.

MDG: A Multi-Task Dynamic Graph Generation Framework for Multivariate Time Series Forecasting

MDG: A Multi-Task Dynamic Graph Generation Framework for Multivariate Time Series Forecasting