Multivariate Time Series Prediction Research Articles

Hybrid Attention Networks for Flow and Pressure Forecasting in Water Distribution Systems

Multivariate geo-sensory time series prediction is challenging because of the complex spatial and temporal correlations. In urban water distribution systems (WDSs), numerous spatial-correlated sensors have been deployed to continuously collect hydraulic data. Forecasts of the monitored flow and pressure time series are of vital importance for operational decision making, alerts, and anomaly detection. To address this issue, we proposed a hybrid dual-stage spatial–temporal attention-based recurrent neural networks (hDS-RNN). Our model consists of two stages: a spatial attention-based encoder and a temporal attention-based decoder. Specifically, a hybrid spatial attention mechanism that employs inputs along the temporal and spatial axes is proposed. Experiments on a real-world data set are conducted, which demonstrate that our model outperformed seven baseline models in flow and pressure predictions in WDS.

Learning Both Dynamic-Shared and Dynamic-Specific Patterns for Chaotic Time-Series Prediction.

In the real world, multivariate time series from the dynamical system are correlated with deterministic relationships. Analyzing them dividedly instead of utilizing the shared-pattern of the dynamical system is time consuming and cumbersome. Multitask learning (MTL) is an effective inductive bias method to utilize latent shared features and discover the structural relationships from related tasks. Base on this concept, we propose a novel MTL model for multivariate chaotic time-series prediction, which could learn both dynamic-shared and dynamic-specific patterns. We implement the dynamic analysis of multiple time series through a special network structure design. The model could disentangle the complex relationships among multivariate chaotic time series and derive the common evolutionary trend of the multivariate chaotic dynamical system by inductive bias. We also develop an efficient Crank-Nicolson-like curvilinear update algorithm based on the alternating direction method of multipliers (ADMM) for the nonconvex nonsmooth Stiefel optimization problem. Simulation results and analysis demonstrate the effectiveness on dynamic-shared pattern discovery and prediction performance.

DTDR–ALSTM: Extracting dynamic time-delays to reconstruct multivariate data for improving attention-based LSTM industrial time series prediction models

Taking advantage of varying degrees of attention on specific features, attention-based long short-term memory (ALSTM) networks have made inroads into the industrial multivariate time series prediction sector recently. However, conventional ALSTM models usually employ static time-delays to constant select input/output pairings of multivariate data, which apparently ignores dynamics of transmission time between industrial process variables and degrades the prediction performance by incorrectly extracting process characteristics. In response to this problem, this paper proposes a novel approach to extracting dynamic time-delays to reconstruct (DTDR) multivariate data for an improved ALSTM prediction model. Therein, the temporal locations and spans of multivariate data are adaptively tailored to input/output pairings of the ALSTM network according to the dynamic time-delays. Specifically, the multivariate data can be accurately matched in temporal positions, and the data information in the original temporal spans with Status transfer time abnormal are replaced. Consequently, this prediction model not only appropriately utilizes dynamics between the predicting and correlated variables, but also makes better attentions on key features extracted from optimum data. Applied to industrial distillation and methanol production processes, the proposed method demonstrates the capability of significantly improving network training speeds as well as prediction accuracies in contrast to static time-delay based ALSTM and LSTM models, expecting even more applications.

Real-Time Prediction of Large-Scale Ship Model Vertical Acceleration Based on Recurrent Neural Network

In marine environments, ships are bound to be disturbed by several external factors, which can cause stochastic fluctuations and strong nonlinearity in the ship motion. Predicting ship motion is pivotal to ensuring ship safety and providing early warning of risks. This report proposes a real-time ship vertical acceleration prediction algorithm based on the long short-term memory (LSTM) and gated recurrent units (GRU) models of a recurrent neural network. The vertical acceleration time history data at the bow, middle, and stern of a large-scale ship model were obtained by performing a self-propulsion test at sea, and the original data were pre-processed by resampling and normalisation via Python. The prediction results revealed that the proposed algorithm could accurately predict the acceleration time history data of the large-scale ship model, and the root mean square error between the predicted and real values was no greater than 0.1. The optimised multivariate time series prediction program could reduce the calculation time by approximately 55% compared to that of a univariate time series prediction program, and the run time of the GRU model was better than that of the LSTM model.

Modified BBO-Based Multivariate Time-Series Prediction System With Feature Subset Selection and Model Parameter Optimization.

Multivariate time-series prediction is a challenging research topic in the field of time-series analysis and modeling, and is continually under research. The echo state network (ESN), a type of efficient recurrent neural network, has been widely used in time-series prediction, but when using ESN, two crucial problems have to be confronted: 1) how to select the optimal subset of input features and 2) how to set the suitable parameters of the model. To solve this problem, the modified biogeography-based optimization ESN (MBBO-ESN) system is proposed for system modeling and multivariate time-series prediction, which can simultaneously achieve feature subset selection and model parameter optimization. The proposed MBBO algorithm is an improved evolutionary algorithm based on biogeography-based optimization (BBO), which utilizes an S -type population migration rate model, a covariance matrix migration strategy, and a Lévy distribution mutation strategy to enhance the rotation invariance and exploration ability. Furthermore, the MBBO algorithm cannot only optimize the key parameters of the ESN model but also uses a hybrid-metric feature selection method to remove the redundancies and distinguish the importance of the input features. Compared with the traditional methods, the proposed MBBO-ESN system can discover the relationship between the input features and the model parameters automatically and make the prediction more accurate. The experimental results on the benchmark and real-world datasets demonstrate that MBBO outperforms the other traditional evolutionary algorithms, and the MBBO-ESN system is more competitive in multivariate time-series prediction than other classic machine-learning models.

A deep learning model to effectively capture mutation information in multivariate time series prediction

In real-world complex multivariate time series data, mutation phenomena can significantly affect variation rules of target series. Meanwhile, there is no specific learning mechanism for the current deep learning model to capture mutation information in time series prediction. To this end, we propose a new deep learning model to capture mutation information between data. To capture the impact of mutation information on target series, a new function mapping is designed in the attention mechanism of the encoder to process the fusion of historical hidden state and cell state information; and an LSTM with transformation mechanism is proposed in the encoder to process the input information flow and learn the mutation information. In addition, an adaptive self-paced curriculum learning mechanism is designed to obtain mutation information that may be ignored among mini-batch samples. Finally, we define an objective function for multivariate time series prediction, which can extract the influence of temporal correlation information and mutation information within the data on target series. Our model can achieve superior performance than all baseline methods on five real-world datasets in different fields.

Online Sequential Model for Multivariate Time Series Prediction With Adaptive Forgetting Factor

In the process of online prediction of multivariable non-stationary time series by kernel extreme learning machine (KELM), the dynamic characteristics of the system which are difficult to determine have always posed a big problem. We propose an online sequential prediction model with an adaptive forgetting factor (AFF) for multivariable time series to solve this problem. The multivariable time series instead of variable itself is reconstructed firstly. AFF is introduced into the objective function and can be adjusted iteratively and adaptively with the system changes. As a result, higher weight can be allocated for the fresh and more important samples while the old failure samples can be quickly forgotten. The model sparsification uses a fast leave-one-out cross-validation (FLOO-CV) method to set a prediction error threshold so that samples can be selected conditionally to form a dictionary. Besides, the dictionary parameters, including AFF and kernel parameters, are recursively updated simultaneously without increasing calculation complexity. The experimental results show that, compared with four fashionable KLEM methods, the proposed AFF-OSKELM has a better dynamic tracking ability and adaptability. Moreover, compared with single variable prediction, the spatial reconstructed multivariable has higher prediction accuracy and stability.

The expressivity and training of deep neural networks: Toward the edge of chaos?

Expressivity is one of the most significant issues in assessing neural networks. In this paper, we provide a quantitative analysis of the expressivity for the deep neural network (DNN) from its dynamic model, where the Hilbert space is employed to analyze the convergence and criticality. We study the feature mapping of several widely used activation functions obtained by Hermite polynomials, and find sharp declines or even saddle points in the feature space, which stagnate the information transfer in DNNs. We then present a new activation function design based on the Hermite polynomials for better utilization of spatial representation. Moreover, we analyze the information transfer of DNNs, emphasizing the convergence problem caused by the mismatch between input and topological structure. We also study the effects of input perturbations and regularization operators on critical expressivity. Our theoretical analysis reveals that DNNs use spatial domains for information representation and evolve to the edge of chaos as depth increases. In actual training, whether a particular network can ultimately arrive the edge of chaos depends on its ability to overcome convergence and pass information to the required network depth. Finally, we demonstrate the empirical performance of the proposed hypothesis via multivariate time series prediction and image classification examples.

Multistage attention network for multivariate time series prediction

The deep learning model has been used to predict the variation rule of the target series of multivariate time series data. Based on the attention mechanism, the influence information of multiple non-predictive time series on target series in different time stages is processed as the same weight in the previous studies. However, on real-world datasets, multiple non-predictive time series has different influence (such as different mutation information) on target series in different time stages. Therefore, a new multistage attention network is designed to capture the different influence. The model is mainly composed of the influential attention mechanism and temporal attention mechanism. In the influential attention mechanism, the same and different time stage attention mechanisms are used to capture the influence information of different non-predictive time series on the target series over time. In the temporal attention mechanism, the variation law of data can be captured over time. Besides, the prediction performance of proposed model on two different real-world multivariate time series datasets is comprehensively evaluated. The results show that, the prediction performance of the proposed model beat all baseline models and SOTA models. In a word, the multistage attention network model can effectively learn the information of the influence of different non-predictive time series on the target series in different time stages in the historical data.

DSTP-RNN: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction

Long-term prediction of multivariate time series is still an important but challenging problem. The key to solve this problem is capturing (1) the spatial correlations at the same time, (2) the spatio-temporal relationships at different times, and (3) long-term dependency of the temporal relationships between different series. Attention-based recurrent neural networks (RNN) can effectively represent and learn the dynamic spatio-temporal relationships between exogenous series and target series, but they only perform well in one-step time prediction and short-term time prediction. In this paper, inspired by human attention mechanism including the dual-stage two-phase (DSTP) model and the influence mechanism of target information and non-target information, we propose DSTP-based RNN (DSTP-RNN) and DSTP-RNN-Ⅱ respectively for long-term time series prediction. Specifically, we first propose the DSTP-based structure to enhance the spatial correlations between exogenous series. The first phase produces violent but decentralized response weight, while the second phase leads to stationary and concentrated response weight. Then, we employ multiple attentions on target series to boost the long-term dependency. Finally, we study the performance of deep spatial attention mechanism and provide interpretation. Experimental results demonstrate that the present work can be successfully used to develop expert or intelligent systems for a wide range of applications, with state-of-the-art performances superior to nine baseline methods on four datasets in the fields of energy, finance, environment and medicine, respectively. Overall, the present work carries a significant value not merely in the domain of machine intelligence and deep learning, but also in the fields of many applications.

Hybrid cycle reservoir with jumps for multivariate time series prediction: industrial application in oil drilling process

Industrial oil drilling processes usually produce high-dimensional multivariate time series data, in which the significant data changes associated with key variables possibly indicate potential drilling accidents. Therefore, it is of great significance to establish a multivariate time series prediction model for the safety of drilling operations. Recently, echo state networks (ESNs) have been widely used in multitime series predictions. However, traditional ESNs use randomly generated sparse network structures and single neuron models, which makes it difficult to achieve a satisfactory performance for complex multivariate time series predictions. In response to this problem, this study proposes a novel hybrid cycle reservoir with jumps (HCRJ) which combines the hybrid wavelet neuron model with the cycle reservoir with jumps (CRJ) structure. In the face of high-dimensional data sets generated by oil drilling, this paper selects a principal component analysis algorithm combined with certain process knowledge to find key variables before related variables are specified by means of Gray correlation analysis methods. The HCRJ networks are used to realize temporal predictions based on data of related variables. To confirm the validity of the model, the HCRJ networks are applied to an oil drilling process and compared to traditional ESNs and CRJ methods. The results show that the HCRJ networks enrich the dynamic characteristics of networks without increasing the complexity of the reserve pool, which helps improve the prediction accuracy of the oil drilling sequence and reduce the occurrence of drilling accidents.

Multivariate Chaotic Time Series Prediction Based on Improved Grey Relational Analysis

In multivariate chaotic time series prediction, correlation analysis is important for reducing input dimensions and improving prediction performance. Grey relational analysis (GRA) has proved to be an effective method for data correlation analysis, especially for inexact data and incomplete data. In GRA, points are usually regarded as objects, and the distance between points or the concave and convex degree are mostly used to measure the correlations. However, with discrete variables, correlation analysis results always tend to have some deviations when using prior GRA methods. Furthermore, GRA methods cannot directly use vector datasets. Therefore, in this paper, an improved GRA method is proposed based on vector projections. The input and output variables are expressed as vectors by linking two adjacent points. The vectors, instants of the points, are regarded as the objects, and the projection length of input variables to output variables is used to measure the correlations. The smaller the difference between the projection length and the input variables, the higher the correlation. Then, a hybrid variable selection and prediction model is proposed based on the improved GRA method for multivariate chaotic time series predictions, in order to overcome the negative effects of irrelevant and redundant variables caused by phase-space reconstruction. The experimental results based on the gas furnace dataset and San Francisco river runoff dataset demonstrate that the improved GRA method is effective for data correlation analysis, and the prediction accuracy is better than prior GRA-based methods.

Non-Stationary Multivariate Time Series Prediction with MIX Gated Unit

Non-Stationary Multivariate Time Series Prediction with MIX Gated Unit

EA-LSTM: Evolutionary attention-based LSTM for time series prediction

Time series prediction with deep learning methods, especially Long Short-term Memory Neural Network (LSTM), have scored significant achievements in recent years. Despite the fact that LSTM can help to capture long-term dependencies, its ability to pay different degree of attention on sub-window feature within multiple time-steps is insufficient. To address this issue, an evolutionary attention-based LSTM training with competitive random search is proposed for multivariate time series prediction. By transferring shared parameters, an evolutionary attention learning approach is introduced to LSTM. Thus, like that for biological evolution, the pattern for importance-based attention sampling can be confirmed during temporal relationship mining. To refrain from being trapped into partial optimization like traditional gradient-based methods, an evolutionary computation inspired competitive random search method is proposed, which can well configure the parameters in the attention layer. Experimental results have illustrated that the proposed model can achieve competetive prediction performance compared with other baseline methods.

Multiple convolutional neural networks for multivariate time series prediction

Multivariate time series prediction, with a profound impact on human social life, has been attracting growing interest in machine learning research. However, the task of time series forecasting is very challenging because it is affected by many complex factors. For example, in predicting traffic and solar power generation, weather can bring great trouble. In particular, for strictly periodic time series, if the periodic information can be extracted from the historical sequence data to the maximum, the accuracy of the prediction will be greatly improved. At present, for time series prediction tasks, the sequence models based on RNN have made great progress. However, the sequence models has difficulty in capturing global information, failing to well highlight the periodic characteristics of the time series. But the this problem can be solved by CNN models. So in this paper, we propose a model called Multiple CNNs to solve the problem of periodic multivariate time series prediction. The working process of Multiple CNNs is analyzing the periodicity of time series, extracting the closeness and the long and short periodic information of the predicted target respectively, and finally integrating the characteristics of the three parts to make the prediction. Moreover, the model is highly flexible, which allows users to freely adjust the cycle span set in the model according to their own data characteristics. Tests on two large real-world datasets, show that our model has a strong advantage over other time series prediction methods.

An intelligent interconnected network with multiple reservoir computing

In this paper, considering the decomposition and composition mechanism of the traditional reservoir, an intelligent interconnected network with multiple reservoir computing in series–parallel configuration, called MRC-IIN, is proposed for multivariate time series classification and prediction. Firstly, according to the unsupervised learning of restricted Boltzmann machine and the Pearson correlation of multivariate inputs, the number of 1st column reservoirs of MRC-IIN can be determined, and each row interconnected reservoirs of MRC-IIN can also be determined. Secondly, according to the input autocorrelation of 1st column reservoirs, the number of each row reservoirs of MRC-IIN can be determined, such that the dynamic characteristics of the corresponding input can be fully reflected. Thirdly, through using the advantages of principal component analysis, each reservoir size of MRC-IIN can be determined, such that the feature information of each row reservoir states can be reduced layer by layer, and the most essential feature information can be retained. Fourthly, in order to ensure the stable application of MRC-IIN, a sufficient condition for global echo state property of MRC-IIN is given. Finally, the simulation results show that the MRC-IIN can improve classification and prediction performance.

Broad echo state network for multivariate time series prediction

In this paper, a broad echo state network with multiple reservoirs in parallel configuration (Broad-ESN) is proposed for a class of multivariate time series prediction. Firstly, through the unsupervised learning algorithm of restricted Boltzmann machine (RBM), the number of reservoirs of Broad-ESN can be determined, such that the dynamic characteristics of a class of multivariate time series can be fully reflected. Secondly, a parameter optimization method based on Davidon–Fletcher–Powell (DFP) quasi-Newton algorithm is proposed to optimize the reservoir parameters of Broad-ESN. Meanwhile, an output weights learning method based on output error is given to train the output weights of Broad-ESN. Thirdly, a sufficient condition for the echo state property of Broad-ESN is given. Finally, four examples are given to verify the effectiveness of Broad-ESN.

Application research on PM2.5 concentration prediction of multivariate chaotic time series

PM2.5 is affected by complex factors such as meteorological elements in the air system and other pollutants in the air. So PM2.5 has chaotic property, which makes the prediction of PM2.5 concentration extremely difficult. In order to improve the prediction accuracy of PM2.5 concentration, this paper introduces the chaotic time series prediction method to establish multivariate time for PM2.5 concentration. The sequence prediction model achieves short-term predictions based on the hour concentration of PM2.5 in Beijing. Firstly, the chaotic time series phase space of the relevant unit is expanded into the multi-time sequence phase space, and the multi-time sequence phase space matrix of PM2.5 concentration is constructed. Then the RBF neural network is used to predict the state point in the multi-phase space system. The phase space points of the PM2.5 concentration sequence is separated for prediction. Finally, the comparison between the prediction model and the traditional prediction model is carried out. The results show that the root mean square error of the predicted PM2.5 concentration in the multivariate chaotic time series prediction model based on the phase space reconstruction is 4.92% in the next 5 hours. The average absolute error is 2.40%, which is more effective than the commonly used statistical prediction method.

Compound Autoregressive Network for Prediction of Multivariate Time Series

The prediction information has effects on the emergency prevention and advanced control in various complex systems. There are obvious nonlinear, nonstationary, and complicated characteristics in the time series. Moreover, multiple variables in the time‐series impact on each other to make the prediction more difficult. Then, a solution of time‐series prediction for the multivariate was explored in this paper. Firstly, a compound neural network framework was designed with the primary and auxiliary networks. The framework attempted to extract the change features of the time series as well as the interactive relation of multiple related variables. Secondly, the structures of the primary and auxiliary networks were studied based on the nonlinear autoregressive model. The learning method was also introduced to obtain the available models. Thirdly, the prediction algorithm was concluded for the time series with multiple variables. Finally, the experiments on environment‐monitoring data were conducted to verify the methods. The results prove that the proposed method can obtain the accurate prediction value in the short term.

Towards forecast techniques for business analysts of large commercial data sets using matrix factorization methods

This research article suggests that there are significant benefits in exposing demand planners to forecasting methods using matrix completion techniques. This study aims to contribute to a better understanding of the field of forecasting with multivariate time series prediction by focusing on the dimension of large commercial data sets with hierarchies. This research highlights that there has neither been sufficient academic research in this sub-field nor dissemination among practitioners in the business sector. This study seeks to innovate by presenting a matrix completion method for short-term demand forecast of time series data on relevant commercial problems. Albeit computing intensive, this method outperforms the state of the art while remaining accessible to business users. The object of research is matrix completion for time series in a big data context within the industry. The subject of the research is forecasting product demand using techniques for multivariate hierarchical time series prediction that are both precise and accessible to non-technical business experts. Apart from a methodological innovation, this research seeks to introduce practitioners to novel methods for hierarchical multivariate time series prediction. The research outcome is of interest for organizations requiring precise forecasts yet lacking the appropriate human capital to develop them.