Multivariate Time Series Forecasting Research Articles

FC-GAGA: Fully Connected Gated Graph Architecture for Spatio-Temporal Traffic Forecasting

Forecasting of multivariate time-series is an important problem that has applications in traffic management, cellular network configuration, and quantitative finance. A special case of the problem arises when there is a graph available that captures the relationships between the time-series. In this paper we propose a novel learning architecture that achieves performance competitive with or better than the best existing algorithms, without requiring knowledge of the graph. The key element of our proposed architecture is the learnable fully connected hard graph gating mechanism that enables the use of the state-of-the-art and highly computationally efficient fully connected time-series forecasting architecture in traffic forecasting applications. Experimental results for two public traffic network datasets illustrate the value of our approach, and ablation studies confirm the importance of each element of the architecture. The code is available here: https://github.com/boreshkinai/fc-gaga.

Temporal Latent Auto-Encoder: A Method for Probabilistic Multivariate Time Series Forecasting

Probabilistic forecasting of high dimensional multivariate time series is a notoriously challenging task, both in terms of computational burden and distribution modeling. Most previous work either makes simple distribution assumptions or abandons modeling cross-series correlations. A promising line of work exploits scalable matrix factorization for latent-space forecasting, but is limited to linear embeddings, unable to model distributions, and not trainable end-to-end when using deep learning forecasting. We introduce a novel temporal latent auto-encoder method which enables nonlinear factorization of multivariate time series, learned end-to-end with a temporal deep learning latent space forecast model. By imposing a probabilistic latent space model, complex distributions of the input series are modeled via the decoder. Extensive experiments demonstrate that our model achieves state-of-the-art performance on many popular multivariate datasets, with gains sometimes as high as 50% for several standard metrics.

Assessing the performance of deep learning models for multivariate probabilistic energy forecasting

Deep learning models have the potential to advance the short-term decision-making of electricity market participants and system operators by capturing the complex dependences and uncertainties of power system operation. Currently, however, the adoption of global deep learning models for multivariate energy forecasting in power systems is far behind the developments in the deep learning research field. In this context, the objectives of this study are to review recent developments in the field of probabilistic, multivariate, and multihorizon time series forecasting and empirically evaluate the performance of novel global deep learning models for forecasting wind and solar generation, electricity load, and wholesale electricity price for intraday and day-ahead time horizons. Two forecast types, deterministic and probabilistic forecasts, are studied. The evaluation data consist of real-world datasets with hourly resolution at the levels of an individual customer and regional and national electricity market bidding zones. The model evaluation criteria include achievable levels of forecasting accuracy and uncertainty risks, hyperparameter sensitivity, the effect of exogenous variables and fieldwise dataset split, and run-time efficiency factors, such as memory utilization, simulation time, electricity consumption, and convergence rate. We conclude that the performance of the global models is more beneficial for intraday forecasts of heterogeneous datasets with nonuniform patterns of time series, but can be affected by the hyperparameter sensitivity and hardware limitations with the growth of dataset dimensionality. The results can serve as a reference point for the quantitative evaluation of deep learning models for probabilistic multivariate energy forecasting in power systems.

Multivariate Time Series Forecasting with Dilated Residual Convolutional Neural Networks for Urban Air Quality Prediction

Multivariate time series (MTS) forecasting is a research field that is gaining more and more importance as time series data generators proliferate in the growing era of Internet of Things. Deep learning architecture for MTS data has been and still a very active research area as there is no comprehensive comparative study of the different architectures, let alone a perfect architecture that can solve all types of time series modeling problem. In this paper, we start by highlighting time series analysis requirements, to propose a new model for conditional multivariate time series forecasting based on the lately introduced WaveNet architecture. Our model is composed of stacked residual causal dilated convolutions that provide large scope in time series history and foster learning of long-term dependencies. Parameterized skip connections allow catching early trends and properties while conditioning enables modeling the associations that exist between multivariate data. We used group normalization for its stability and independence from the batch size. We present a deep comparison of the structure, flexibility, flow control, memory consumption, robustness and stability capacities of our new model against those of recurrent neural networks state-of-the-art approaches—long-short-term-memory (LSTM) and gated recurrent unit (GRU). To assess the performances of our model, we conduct extensive experimental testing on the task of urban air quality prediction in Marrakesh city using six real-world multi-sensor and multivariate time series datasets. We compare the results of our model against the results of LSTM and GRU. Our experiments revealed that our proposed WaveNet-temporal-CNN outperform recurrent models ability to learn long-term dependencies in a time-efficient way.

Energy time series forecasting-analytical and empirical assessment of conventional and machine learning models

Machine learning methods have been adopted in the literature as contenders to conventional methods to solve the energy time series forecasting (TSF) problems. Recently, deep learning methods have been emerged in the artificial intelligence field attaining astonishing performance in a wide range of applications. Yet, the evidence about their performance in to solve the energy TSF problems, in terms of accuracy and computational requirements, is scanty. Most of the review articles that handle the energy TSF problem are systematic reviews, however, a qualitative and quantitative study for the energy TSF problem is not yet available in the literature. The purpose of this paper is twofold, first it provides a comprehensive analytical assessment for conventional, machine learning, and deep learning methods that can be utilized to solve various energy TSF problems. Second, the paper carries out an empirical assessment for many selected methods through three real-world datasets. These datasets related to electrical energy consumption problem, natural gas problem, and electric power consumption of an individual household problem. The first two problems are univariate TSF and the third problem is a multivariate TSF. Compared to both conventional and machine learning contenders, the deep learning methods attain a significant improvement in terms of accuracy and forecasting horizons examined. In the meantime, their computational requirements are notably greater than other contenders. Eventually, the paper identifies a number of challenges, potential research directions, and recommendations to the research community may serve as a basis for further research in the energy forecasting domain.

Multivariate time series prediction of high dimensional data based on deep reinforcement learning

In order to improve the prediction accuracy of high-dimensional data time series, a high-dimensional data multivariate time series prediction method based on deep reinforcement learning is proposed. The deep reinforcement learning method is used to solve the time delay of each variable and mine the data characteristics. According to the principle of maximum conditional entropy, the embedding dimension of the phase space is expanded, and a multivariate time series model of high-dimensional data is constructed. Thus, the conversion of reconstructed coordinates from low-dimensional to high-dimensional can be kept relatively stable. The strong independence and low redundancy of the final reconstructed phase space construct an effective model input vector for multivariate time series forecasting. Numerical experiments of classical multivariable chaotic time series show that the method proposed in this paper has better forecasting effect, which shows the forecasting effectiveness of this method.

PLS-CNN-BiLSTM: An End-to-End Algorithm-Based Savitzky–Golay Smoothing and Evolution Strategy for Load Forecasting

This paper proposes an effective deep learning framework for Short-Term Load Forecasting (STLF) of multivariate time series. The proposed model consists of a hybrid Convolutional neural network-Bidirectional Long Short-Term Memory (CBiLSTM) based on the Evolution Strategy (ES) method and the Savitzky–Golay (SG) filter (SG-CBiLSTM). The adopted methodology incorporates the virtue of different prepossessing blocks to enhance the performance of the CBiLSTM model. In particular, a data-augmentation strategy is employed to synthetically improve the feature representation of the CBiLSTM model. The augmented data is forwarded to the Partial Least Square (PLS) method to select the most informative features above the predefined threshold. Next, the SG algorithm is computed for smoothing the load to enhance the learning capabilities of the underlying system. The structure of the SG-CBiLSTM for the ISO New England dataset is optimized using the ES technique. Finally, the CBiLSTM model generates output forecasts. The proposed approach demonstrates a remarkable improvement in the performance of the original CBiLSTM model. Furthermore, the experimental results strongly confirm the high effectiveness of the proposed SG-CBiLSTM model compared to the state-of-the-art techniques.

LAVARNET: Neural network modeling of causal variable relationships for multivariate time series forecasting

Multivariate time series forecasting is of great importance to many scientific disciplines and industrial sectors. The evolution of a multivariate time series depends on the dynamics of its variables and the connectivity network of causal interrelationships among them. Most of the existing time series models do not account for the causal effects among the system’s variables and even if they do, they rely just on determining the between-variables causality network. Knowing the structure of such a complex network, and even more specifically knowing the exact lagged variables that contribute to the underlying process is crucial for the task of multivariate time series forecasting. The latter is a rather unexplored source of information to leverage. In this direction, here a novel neural network-based architecture is proposed, termed LAgged VAriable Representation NETwork (LAVARNET), which intrinsically estimates the importance of lagged variables and combines high dimensional latent representations of them to predict future values of time series. Our model is compared with other baseline and state of the art neural network architectures on one simulated data set and four real data sets from the domains of meteorology, music, solar activity, and finance. The proposed architecture outperforms the competitive architectures in most of the experiments.

Deep Fuzzy Cognitive Maps for Interpretable Multivariate Time Series Prediction

The fuzzy cognitive map (FCM) is a powerful model for system state prediction and interpretable knowledge representation. Recent years have witnessed the tremendous efforts devoted to enhancing the basic FCM, such as introducing temporal factors, uncertainty or fuzzy rules to improve interpretation, and introducing fuzzy neural networks or wavelets to improve time series prediction. But how to achieve high-precision yet interpretable prediction in cross-domain real-life applications remains a great challenge. In this article, we propose a novel FCM extension called deep FCM (DFCM) for multivariate time series forecasting, in order to take both the advantage of FCM in interpretation and the advantage of deep neural networks in prediction. Specifically, to improve the predictive power, DFCM leverages a fully connected neural network to model connections (relationships) among concepts in a system, and a recurrent neural network to model unknown exogenous factors that have influences on system dynamics. Moreover, to foster model interpretability encumbered by the embedded deep structures, a partial derivative-based approach is proposed to measure the connection strengths between concepts in DFCM. An alternate function gradient descent algorithm is then proposed for parameter inference. The effectiveness of DFCM is validated over four publicly available datasets with the presence of seven baselines. DFCM indeed provides an important clue to building interpretable predictors for real-life applications.

Towards Better Forecasting by Fusing Near and Distant Future Visions

Multivariate time series forecasting is an important yet challenging problem in machine learning. Most existing approaches only forecast the series value of one future moment, ignoring the interactions between predictions of future moments with different temporal distance. Such a deficiency probably prevents the model from getting enough information about the future, thus limiting the forecasting accuracy. To address this problem, we propose Multi-Level Construal Neural Network (MLCNN), a novel multi-task deep learning framework. Inspired by the Construal Level Theory of psychology, this model aims to improve the predictive performance by fusing forecasting information (i.e., future visions) of different future time. We first use the Convolution Neural Network to extract multi-level abstract representations of the raw data for near and distant future predictions. We then model the interplay between multiple predictive tasks and fuse their future visions through a modified Encoder-Decoder architecture. Finally, we combine traditional Autoregression model with the neural network to solve the scale insensitive problem. Experiments on three real-world datasets show that our method achieves statistically significant improvements compared to the most state-of-the-art baseline methods, with average 4.59% reduction on RMSE metric and average 6.87% reduction on MAE metric.

Joint Modeling of Local and Global Temporal Dynamics for Multivariate Time Series Forecasting with Missing Values

Multivariate time series (MTS) forecasting is widely used in various domains, such as meteorology and traffic. Due to limitations on data collection, transmission, and storage, real-world MTS data usually contains missing values, making it infeasible to apply existing MTS forecasting models such as linear regression and recurrent neural networks. Though many efforts have been devoted to this problem, most of them solely rely on local dependencies for imputing missing values, which ignores global temporal dynamics. Local dependencies/patterns would become less useful when the missing ratio is high, or the data have consecutive missing values; while exploring global patterns can alleviate such problem. Thus, jointly modeling local and global temporal dynamics is very promising for MTS forecasting with missing values. However, work in this direction is rather limited. Therefore, we study a novel problem of MTS forecasting with missing values by jointly exploring local and global temporal dynamics. We propose a new framework øurs, which leverages memory network to explore global patterns given estimations from local perspectives. We further introduce adversarial training to enhance the modeling of global temporal distribution. Experimental results on real-world datasets show the effectiveness of øurs for MTS forecasting with missing values and its robustness under various missing ratios.

Partial Correlation-Based Attention for Multivariate Time Series Forecasting

A multivariate time-series forecasting has great potentials in various domains. However, it is challenging to find dependency structure among the time-series variables and appropriate time-lags for each variable, which change dynamically over time. In this study, I suggest partial correlation-based attention mechanism which overcomes the shortcomings of existing pair-wise comparisons-based attention mechanisms. Moreover, I propose data-driven series-wise multi-resolution convolutional layers to represent the input time-series data for domain agnostic learning.

Feature selection based multivariate time series forecasting: An application to antibiotic resistance outbreaks prediction.

Antimicrobial resistance has become one of the most important health problems and global action plans have been proposed globally. Prevention plays a key role in these actions plan and, in this context, we propose the use of Artificial Intelligence, specifically Time Series Forecasting techniques, for predicting future outbreaks of Methicillin-resistant Staphylococcus aureus (MRSA). Infection incidence forecasting is approached as a Feature Selection based Time Series Forecasting problem using multivariate time series composed of incidence of Staphylococcus aureus Methicillin-sensible and MRSA infections, influenza incidence and total days of therapy of both of Levofloxacin and Oseltamivir antimicrobials. Data were collected from the University Hospital of Getafe (Spain) from January 2009 to January 2018, using months as time granularity. The main contributions of the work are the following: the applications of wrapper feature selection methods where the search strategy is based on multi-objective evolutionary algorithms (MOEA) along with evaluators based on the most powerful state-of-the-art regression algorithms. The performance of the feature selection methods has been measured using the root mean square error (RMSE) and mean absolute error (MAE) performance metrics. A novel multi-criteria decision-making process is proposed in order to select the most satisfactory forecasting model, using the metrics previously mentioned, as well as the slopes of model prediction lines in the 1, 2 and 3 steps-ahead predictions. The multi-criteria decision-making process is applied to the best models resulting from a ranking of databases and regression algorithms obtained through multiple statistical tests. Finally, to the best of our knowledge, this is the first time that a feature selection based multivariate time series methodology is proposed for antibiotic resistance forecasting. Final results show that the best model according to the proposed multi-criteria decision making process provides a RMSE = (0.1349, 0.1304, 0.1325) and a MAE = (0.1003, 0.096, 0.0987) for 1, 2, and 3 steps-ahead predictions.

A Hybrid Method With Adaptive Sub-Series Clustering and Attention-Based Stacked Residual LSTMs for Multivariate Time Series Forecasting

Multivariate Time Series Forecasting (MTSF) has recently emerged its growing importance in many industries. However, how to reduce the influence of the noise components existing in time series on prediction and extract features effectively are still two challenges in MTSF. This paper focuses on those two challenges and proposes a new prediction method based on decomposition-ensemble framework called adaptive sub-series clustering-stacked residual LSTMs-multi-level attention mechanism (ASC-SRLSTMs-MLAttn). The method consists of three stages: decomposition, prediction, and ensemble. In the decomposition stage, the target series is decomposed by Ensemble Empirical Mode Decomposition into multiple sub-series, which will be clustered and reconstructed by the ASC algorithm to reduce the complexity and the time consumption of prediction. In the prediction stage, the sub-series and correlation series will be fed into SRLSTMs-MLAttn for sub-series prediction. The model is based on the encoder-decoder architecture with stacked residual LSTMs as the encoder, which can effectively capture the dependencies among multi variables and the temporal features from multivariate time series. Besides, a multi-level attention mechanism (MLAttn), which makes full use of the encoding information of the encoder, has been introduced to further improve the prediction performance of the model. In the ensemble stage, the predicted values of each sub-series will be summed to obtain the final prediction of the target time series. We also demonstrate the superiority and effectiveness of our proposed method on four public datasets via the conducted comparison experiment and ablation study.

Discussion of “Bayesian forecasting of multivariate time series: scalability, structure uncertainty and decisions”

The author focuses on the “decoupling and recoupling” idea that can critically increase both computational and forecasting efficiencies in practical problems for economic and financial data. My discussion is twofold. First, I briefly describe the idea with an example of time-varying vector autoregressions, which are widely used in the context. Second, I highlight the issue of how to assess patterns of simultaneous relationships.

Bayesian forecasting of multivariate time series: scalability, structure uncertainty and decisions

I overview recent research advances in Bayesian state-space modeling of multivariate time series. A main focus is on the decouple/recouple concept that enables application of state-space models to increasingly large-scale data, applying to continuous or discrete time series outcomes. The scope includes large-scale dynamic graphical models for forecasting and multivariate volatility analysis in areas such as economics and finance, multi-scale approaches for forecasting discrete/count time series in areas such as commercial sales and demand forecasting, and dynamic network flow models for areas including internet traffic monitoring. In applications, explicit forecasting, monitoring and decision goals are paramount and should factor into model assessment and comparison, a perspective that is highlighted.

Reply to Discussion of “Bayesian forecasting of multivariate time series: scalability, structure uncertainty and decisions”

Reply to Discussion of “Bayesian forecasting of multivariate time series: scalability, structure uncertainty and decisions”

Discussion of “Bayesian forecasting of multivariate time series: scalability, structure uncertainty and decisions”

Discussion of “Bayesian forecasting of multivariate time series: scalability, structure uncertainty and decisions”

Multivariate Bayesian Predictive Synthesis in Macroeconomic Forecasting

We present new methodology and a case study in use of a class of Bayesian predictive synthesis (BPS) models for multivariate time series forecasting. This extends the foundational BPS framework to the multivariate setting, with detailed application in the topical and challenging context of multistep macroeconomic forecasting in a monetary policy setting. BPS evaluates—sequentially and adaptively over time—varying forecast biases and facets of miscalibration of individual forecast densities for multiple time series, and—critically—their time-varying inter-dependencies. We define BPS methodology for a new class of dynamic multivariate latent factor models implied by BPS theory. Structured dynamic latent factor BPS is here motivated by the application context—sequential forecasting of multiple U.S. macroeconomic time series with forecasts generated from several traditional econometric time series models. The case study highlights the potential of BPS to improve of forecasts of multiple series at multiple forecast horizons, and its use in learning dynamic relationships among forecasting models or agents.Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

Multivariate Temporal Convolutional Network: A Deep Neural Networks Approach for Multivariate Time Series Forecasting

Multivariable time series prediction has been widely studied in power energy, aerology, meteorology, finance, transportation, etc. Traditional modeling methods have complex patterns and are inefficient to capture long-term multivariate dependencies of data for desired forecasting accuracy. To address such concerns, various deep learning models based on Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) methods are proposed. To improve the prediction accuracy and minimize the multivariate time series data dependence for aperiodic data, in this article, Beijing PM2.5 and ISO-NE Dataset are analyzed by a novel Multivariate Temporal Convolution Network (M-TCN) model. In this model, multi-variable time series prediction is constructed as a sequence-to-sequence scenario for non-periodic datasets. The multichannel residual blocks in parallel with asymmetric structure based on deep convolution neural network is proposed. The results are compared with rich competitive algorithms of long short term memory (LSTM), convolutional LSTM (ConvLSTM), Temporal Convolution Network (TCN) and Multivariate Attention LSTM-FCN (MALSTM-FCN), which indicate significant improvement of prediction accuracy, robust and generalization of our model.