Long-term Time Series Forecasting Research Articles

MDCNet: Long-term time series forecasting with mode decomposition and 2D convolution

Long-term time series forecasting is widely used in various real-world applications, such as weather, traffic, energy, healthcare, etc. Recently, time series decomposition techniques have been adopted in many mainstream forecasting models, such as the prevalent Transformer-based models, to help capture sophisticated temporal patterns and achieve success in several benchmark tasks. However, conventional decomposition algorithms are often based on simple operations or limited to specific fields, and therefore are not effective and applicable enough, especially for complex time series. In this paper, we propose Mode Decomposition and 2D Convolutional Network (MDCNet), a structure-simple yet effective forecasting architecture based on a more effective decomposition method and a multi-frequency time series feature extraction network with multi-scale 2D convolution. Specifically, we first introduce a Variational Mode Decomposition Block to discover intricate time patterns, which decompose time series into trend components and stationary modal components at different main frequencies. Then, we design a Trend Prediction Block and an Intrinsic Mode Functions Prediction Block to capture global correlation and hidden dependencies within different main frequencies, respectively. Furthermore, a Frequency Enhancement Module is designed to further reduce the impact of noise in long-term time series. Experiments on eight benchmark datasets show that MDCNet significantly reduces the error of the previous state-of-the-art method by 15.1% and 11.5% for multivariate and univariate time series, respectively.

MEAformer: An all-MLP transformer with temporal external attention for long-term time series forecasting

Transformer-based models have significantly improved performance in Long-term Time Series Forecasting (LTSF). These models employ various self-attention mechanisms to discover long-term dependencies. However, the computational efficiency is hampered by the inherent permutation invariance of self-attention, and they primarily focus on relationships within the sequence while neglecting potential relationships between different sample sequences. This limits the ability and flexibility of self-attention in LTSF. In addition, the Transformer's decoder outputs sequences in an autoregressive manner, leading to slow inference speed and error accumulation effects, especially for LTSF. Regarding the issues with Transformer-based models for LTSF, we propose a model better suited for LTSF, named MEAformer. MEAformer adopts a fully connected Multi-Layer Perceptron (MLP) architecture consisting of two types of layers: encoder layers and MLP layers. Unlike most encoder layers in Transformer-based models, the MEAformer replaces self-attention with temporal external attention. Temporal external attention explores potential relationships between different sample sequences in the training dataset. Compared to the quadratic complexity of self-attention mechanisms, temporal external attention has efficient linear complexity. Encoder layers can be stacked multiple times to capture time-dependent relationships at different scales. Furthermore, the MEAformer replaces the intricate decoder layers of the original model with more straightforward MLP layers. This modification aims to enhance inference speed and facilitate single-pass sequence generation, effectively mitigating the problem of error accumulation effects. Regarding long-term forecasting, MEAformer achieves state-of-the-art performance on six benchmark datasets, covering five real-world domains: energy, transportation, economy, weather, and disease. Code is available at: https://github.com/huangsiyuan924/MEAformer.

EasyTS: The Express Lane to Long Time Series Forecasting

Responding to the escalating interest in long-term forecasting within the industry, we introduce EasyTS, a comprehensive toolkit engineered to streamline data collection, analysis, and model creation procedures. EasyTS acts as a unified solution, driving progress in long-term time series forecasting. The platform provides effortless access to various time series datasets, including a newly open-sourced multi-scenario dataset in the electricity domain. Integrated visualization and analysis tools help unveil inherent data features and relationships. EasyTS facilitates a user-friendly model validation approach with versatile evaluation criteria. This toolkit allows researchers to compare their models proficiently against renowned benchmarks. With our ongoing commitment to expanding our dataset collection and enhancing toolkit functionalities, we aspire to contribute significantly to the time series forecasting domain. Code is available at this repository: https://github.com/EdgeBigBang/EasyTS.git.

Learning from Polar Representation: An Extreme-Adaptive Model for Long-Term Time Series Forecasting

In the hydrology field, time series forecasting is crucial for efficient water resource management, improving flood and drought control and increasing the safety and quality of life for the general population. However, predicting long-term streamflow is a complex task due to the presence of extreme events. It requires the capture of long-range dependencies and the modeling of rare but important extreme values. Existing approaches often struggle to tackle these dual challenges simultaneously. In this paper, we specifically delve into these issues and propose Distance-weighted Auto-regularized Neural network (DAN), a novel extreme-adaptive model for long-range forecasting of stremflow enhanced by polar representation learning. DAN utilizes a distance-weighted multi-loss mechanism and stackable blocks to dynamically refine indicator sequences from exogenous data, while also being able to handle uni-variate time-series by employing Gaussian Mixture probability modeling to improve robustness to severe events. We also introduce Kruskal-Wallis sampling and gate control vectors to handle imbalanced extreme data. On four real-life hydrologic streamflow datasets, we demonstrate that DAN significantly outperforms both state-of-the-art hydrologic time series prediction methods and general methods designed for long-term time series prediction.

Automated Machine Learning Algorithm Using Recurrent Neural Network to Perform Long-Term Time Series Forecasting

Automated Machine Learning Algorithm Using Recurrent Neural Network to Perform Long-Term Time Series Forecasting

System Identification with Fourier Transformation for Long-term Time Series Forecasting

System Identification with Fourier Transformation for Long-term Time Series Forecasting

Hidformer: Hierarchical dual-tower transformer using multi-scale mergence for long-term time series forecasting

Long-term time series forecasting has received a lot of popularity because of its great practicality. It is also an extremely challenging task since it requires using limited observations to predict values in the long future accurately. Recent works have demonstrated that Transformer has strong potential for this task. However, the permutation-invariant property of the Transformer and some other prominent shortcomings in the current Transformer-based models, such as missing multi-scale local features and information from the frequency domain, significantly limit their performance. To improve the accuracy of the long-term time series forecasting, we propose a Transformer-based model called Hidformer. This model can either learn temporal dynamics from the time domain or discover particular patterns from the frequency domain. We also design a segment-and-merge architecture to provide semantic meanings for the inputs and help the model capture multi-scale local features. Besides, we replace Transformer's multi-head attention with highly-efficient recurrence and linear attention, which gives our model an advantage over other Transformer-based models in terms of computational efficiency. Extensive experiments are conducted on seven real-world benchmarks to verify the effectiveness of Hidformer. The experimental results show that Hidformer achieves 72 top-1 and 69 top-2 scores out of 88 configurations. It dramatically improves the prediction accuracy and outperforms the previous state-of-the-art, proving the superiority of our proposed method.

NFIG-X: Nonlinear Fuzzy Information Granule Series for Long-Term Traffic Flow Time-Series Forecasting

Long-term time series forecasting is an extensive research topic and is of great significance in many fields. However, the task of long-term time series forecasting is accompanied by the problem of increasing cumulative error and decreasing time correlation. To overcome these shortcomings, this work proposes a prediction framework based on non-linear fuzzy information granule (NFIG) series, which can boost the long-term performance of most predictors. Firstly, we propose the representation of NFIG for the first time, replacing the linear core lines with non-linear time-dependent curves. Secondly, we propose a temporal window splitting algorithm based on curvature equations and weighted directed graphs, which can not only merge temporal windows with the same trend but also cointegrate incremental data. Finally, the non-linear trend fuzzy granulation can be employed as a data preprocessing module for various time series predictors to achieve better long-term forecasting performance. As a typical time series forecasting task, the precise long-term forecast of traffic flow data can relieve the overburdened traffic system and improve the traffic environment to a certain extent. Thus, the proposed method is employed for long-term traffic flow forecasting. Compared with existing forecasting models, which achieves superior performances.

CrossWaveNet: A dual-channel network with deep cross-decomposition for Long-term Time Series Forecasting

Many real-world applications require predicting data changes over a longer period, such as early warning for energy consumption and long-term planning for transportation. Long-term Time Series Forecasting (LTSF) demands models with high predictive capability. Recently, transformer-based models have achieved notable success in LTSF. However, the inherent nature of the permutation-invariant self-attention mechanism in Transformers can lead to the loss of certain temporal patterns. Therefore, we propose a dual-channel network with deep cross-decomposition for LTSF, called CrossWaveNet. In order to better capture long-term dependencies, unlike the complex network structure of Transformer, a simpler and more effective dual-channel network model for season and trend-cyclical is constructed. The original time series is decomposed into seasonal and trend-cyclical components simultaneously, and the dual-channel network structure is used to extract their features, respectively. This structure can improve the model’s generalization ability, which can have good results on various types of time series data. To effectively improve the accuracy of data decomposition, the moving average method is first used to obtain coarse-grained seasonal and trend-cyclical components, and then the extracted seasonal and trend-cyclical components are gradually merged into their respective channels through the dual-channel network with deep cross-decomposition, thereby obtaining fine-grained components and effectively improving prediction performance. Compared with the state-of-the-art methods in long-term forecasting, CrossWaveNet has achieved the highest prediction accuracy and significant relative performance improvement in fields such as energy, transportation, weather, and disease transmission.

DBAFormer: A Double-Branch Attention Transformer for Long-Term Time Series Forecasting

The transformer-based approach excels in long-term series forecasting. These models leverage stacking structures and self-attention mechanisms, enabling them to effectively model dependencies in series data. While some approaches prioritize sparse attention to tackle the quadratic time complexity of self-attention, it can limit information utilization. We introduce a creative double-branch attention mechanism that simultaneously captures intricate dependencies in both temporal and variable perspectives. Moreover, we propose query-independent attention, taking into account the near-identical attention allocated by self-attention to different query positions. This enhances efficiency and reduces the impact of redundant information. We integrate the double-branch query-independent attention into popular transformer-based methods like Informer, Autoformer, and Non-stationary transformer. The results obtained from conducting experiments on six practical benchmarks consistently validate that our novel attention mechanism substantially improves the long-term series forecasting performance in contrast to the baseline approach.

Are Transformers Effective for Time Series Forecasting?

Recently, there has been a surge of Transformer-based solutions for the long-term time series forecasting (LTSF) task. Despite the growing performance over the past few years, we question the validity of this line of research in this work. Specifically, Transformers is arguably the most successful solution to extract the semantic correlations among the elements in a long sequence. However, in time series modeling, we are to extract the temporal relations in an ordered set of continuous points. While employing positional encoding and using tokens to embed sub-series in Transformers facilitate preserving some ordering information, the nature of the permutation-invariant self-attention mechanism inevitably results in temporal information loss. To validate our claim, we introduce a set of embarrassingly simple one-layer linear models named LTSF-Linear for comparison. Experimental results on nine real-life datasets show that LTSF-Linear surprisingly outperforms existing sophisticated Transformer-based LTSF models in all cases, and often by a large margin. Moreover, we conduct comprehensive empirical studies to explore the impacts of various design elements of LTSF models on their temporal relation extraction capability. We hope this surprising finding opens up new research directions for the LTSF task. We also advocate revisiting the validity of Transformer-based solutions for other time series analysis tasks (e.g., anomaly detection) in the future.

InParformer: Evolutionary Decomposition Transformers with Interactive Parallel Attention for Long-Term Time Series Forecasting

Long-term time series forecasting (LTSF) provides substantial benefits for numerous real-world applications, whereas places essential demands on the model capacity to capture long-range dependencies. Recent Transformer-based models have significantly improved LTSF performance. It is worth noting that Transformer with the self-attention mechanism was originally proposed to model language sequences whose tokens (i.e., words) are discrete and highly semantic. However, unlike language sequences, most time series are sequential and continuous numeric points. Time steps with temporal redundancy are weakly semantic, and only leveraging time-domain tokens is hard to depict the overall properties of time series (e.g., the overall trend and periodic variations). To address these problems, we propose a novel Transformer-based forecasting model named InParformer with an Interactive Parallel Attention (InPar Attention) mechanism. The InPar Attention is proposed to learn long-range dependencies comprehensively in both frequency and time domains. To improve its learning capacity and efficiency, we further design several mechanisms, including query selection, key-value pair compression, and recombination. Moreover, InParformer is constructed with evolutionary seasonal-trend decomposition modules to enhance intricate temporal pattern extraction. Extensive experiments on six real-world benchmarks show that InParformer outperforms the state-of-the-art forecasting Transformers.

A WOA-CNN-BiLSTM-based multi-feature classification prediction model for smart grid financial markets

Introduction: Smart grid financial market forecasting is an important topic in deep learning. The traditional LSTM network is widely used in time series forecasting because of its ability to model and forecast time series data. However, in long-term time series forecasting, the lack of historical data may lead to a decline in forecasting performance. This is a difficult problem for traditional LSTM networks to overcome.Methods: In this paper, we propose a new deep-learning model to address this problem. This WOA-CNN-BiLSTM model combines bidirectional long short-term memory network BiLSTM and convolution Advantages of Neural Network CNN. We replace the traditional LSTM network with a bidirectional long short-term memory network, BiLSTM, to exploit its ability in capturing long-term dependencies. It can capture long-term dependencies in time series and is bidirectional modelling. At the same time, we use a convolutional neural network (CNN) to extract features of time series data to better represent and capture patterns and regularity in the data. This method combining BiLSTM and CNN can learn the characteristics of time series data more comprehensively, thus improving the accuracy of prediction. Then,to further improve the performance of the CNN-BiLSTM model, we optimize the model using the whale algorithm WOA. This algorithm is a new optimization algorithm, which has good global search ability and convergence speed, and can complete the optimization of the model in a short time.Results: Optimizing the CNN-BiLSTM model through the WOA algorithm can reduce its calculation and training speed, improve the prediction accuracy of the smart grid financial market, and improve the prediction ability of the smart grid financial market. Experimental results show that our proposed CNN-BiLSTM model has better prediction accuracy than other models and can effectively deal with the problem of missing historical data in long-term sequence forecasting.Discussion: This provides necessary help for the development of smart grid financial markets and risk management services, and can promote the development and growth of the smart grid industry. Our research results are of great significance in deep learning, and provide an effective method and idea for solving the financial market forecasting problem of smart grid.

FV-MgNet: Fully connected V-cycle MgNet for interpretable time series forecasting

By investigating iterative methods for a constrained linear model, we propose a new class of fully connected V-cycle MgNet for long-term time series forecasting, which is one of the most difficult tasks in forecasting. MgNet is a CNN model that was proposed for image classification based on the multigrid (MG) methods for solving discretized partial differential equations (PDEs). We replace the convolutional operations with fully connected operations in the existing MgNet and then apply them to forecasting problems. Motivated by the V-cycle structure in MG, we further propose the FV-MgNet, a V-cycle version of the fully connected MgNet, to extract features hierarchically. By evaluating the performance of FV-MgNet on popular datasets and comparing it with state-of-the-art models, we show that the FV-MgNet achieves better results with less memory usage and faster inference speed. In addition, we develop ablation experiments to demonstrate that the structure of FV-MgNet is the best choice among the many variants.

CLformer: Locally grouped auto-correlation and convolutional transformer for long-term multivariate time series forecasting

Improving the performance of long-term time series forecasting is important for real-world applications. Recently, Transformer-based models have achieved significant performance gains in long-term time series prediction. However, these models are memory-intensive and cannot capture temporal patterns at multiple scales. To this end, we propose to integrate the time series decomposition method in the Transformer framework to enable the model to extract short- and long-term time patterns in more predictable seasonal and trend components. In this paper, we propose a Transformer-based model named CLformer. Different from previous methods, we exploit dilated convolutional networks to capture and refine multiple temporally repeated patterns in time series before time series decomposition. To enable the model to capture the dependencies at multiple scales, we propose a local group autocorrelation (LGAC) mechanism. The LGAC mechanism calculates autocorrelation within time series segments, strengthening the model’s ability to capture the local temporal dynamics of series. The stacking of multiple LGAC layers enables the model to capture multi-scale dependencies, which in turn improves the model’s predictive performance. The CLformer outperforms models using the global autocorrelation mechanism and self-attention in both efficiency and accuracy. Experimental results on six benchmark datasets show that our model obtains a relative performance improvement of 11.75% compared to the state-of-the-art methods. In addition, CLformer achieves a relative performance improvement of 18.89% on two datasets without apparent periodicity, demonstrating the effectiveness of our model on time series without significant periodicity.

The Trend-Fuzzy-Granulation-Based Adaptive Fuzzy Cognitive Map for Long-Term Time Series Forecasting

One drawback of using the existing one-step forecasting models for long-term time series prediction is the cumulative errors caused by iterations. In order to overcome this shortcoming, this article proposes a trend-fuzzy-granulation-based adaptive fuzzy cognitive map (FCM) for long-term time series forecasting. Different from the original FCM-based forecasting models, a class of trend fuzzy information granules is built to represent the trend, fluctuation range, and trend persistence of various segments of time series, which are more instrumental and comprehensive than simple magnitude information. Thus, the proposed forecasting model is a granular model according to the form of its inputs and outputs. In an original FCM-based forecasting model, the causal relationships among concepts remain unchanged throughout the training of the whole dataset, however, in reality, the causal relationships may change with the state of concepts. Therefore, it is unreasonable to use the invariable causal relationships which often result in poor predictions. In view of this, we construct an adaptive FCM where different causal relationships are built to forecast concepts of different states. This is the first time to forecast trend fuzzy information granules using an adaptive FCM. Compared with the existing classical forecasting models, the proposed forecasting model achieves superior performance which is verified through a series of experimental studies.

Query Selector–Efficient transformer with sparse attention

We present Query Selector - a sparse attention Transformer algorithm especially efficient for long-term time series forecasting. Time series forecasting is an old and important area of statistical research with vast practical applications to solving real life problems. In recent years, there has been growing interest in applying Deep Learning algorithms to time series forecasting and modeling. However, the best performing Deep Learning algorithm, Transformer, shows some problems with modeling long term time series due to an inherent weakness of this solution i.e. self-attention mechanism. Memory requirements of the canonical Transformer architecture are growing quadratically with sequence length. Query Selector sparse attention Transformer is an attempt to address this problem.

Handling Massive Proportion of Missing Labels in Multivariate Long-Term Time Series Forecasting

Training Deep Learning (DL) models with missing labels is a challenge in diverse engineering applications. Missing value imputation methods have been proposed to try to address this problem, but their performance is affected with Massive Proportion of Missing Labels (MPML). This paper presents a approach for handling MPML in Multivariate Long-Term Time Series Forecasting. It is an two-step process where interpolation (using Gaussian Processes Regression (GPR) and domain knowledge from experts) and prediction model are separated to enable the integration of prior domain knowledge. First, a set of samples of the possible interpolation of the missing outputs are generated by the GPR based on the domain knowledge. Second, the observed input sensor data and interpolated labels from GPR are used to train the prediction model. We evaluated our approach with the development of a soft-sensor with one real datasets to forecast the biomass during recombinant adeno-associated virus (rAAV) production in bioreactors. Our experimental results demonstrate the potential of the approach through quantitative evaluation of the generated forecasts in a case that would be extremely difficult to train a DL model due to MPML.

A deep granular network with adaptive unequal-length granulation strategy for long-term time series forecasting and its industrial applications

An accurate long-term forecasting for some time series in industrial production is substantially significant for improving the economic efficiency of industry enterprise. In this study, a granular computing (GrC)-based deep learning framework is proposed for long-term time series forecasting, which consists of two stages i.e., the adaptive data granulation and the GrC-based deep model construction. In the first stage, for automatically generating the basic information granules with unequal time span adaptively from data, a stacked sparse auto-encoders granulation network is designed, in which the starting and ending points of a granule are adaptively determined by setting a single neuron in the last hidden layer after multi-layer feature extraction. Then, the second stage sees a definition of a partially overlapping sub-block basis (POSB) matrix to extract the features of these granules, based on which a deep sparse coding feature decomposition-based long-term forecasting model is constructed to transform the unequal-length granules into a product of a POSB matrix and a coefficient matrix layer by layer to serve for forecasting. To verify the effectiveness of the proposed method, two synthetic datasets, two real-world datasets and two practical industrial datasets are employed. The experimental results demonstrate that the proposed method outperforms other data-driven ones on long-term time series forecasting, particularly in an industrial case.

Adaptive Temporal-Frequency Network for Time-Series Forecasting

A novel adaptive temporal-frequency network (ATFN), which is an end-to-end hybrid model incorporating deep learning networks and frequency patterns, is proposed for mid- and long-term time series forecasting. Within the framework of the ATFN, an augmented sequence to sequence model is used to learn the trend feature of complicated nonstationary time series, a frequency-domain block is used to capture dynamic and complicated periodic patterns of time series data, and a fully connected neural network is used to combine the trend and periodic features for producing a final forecast. An adaptive frequency mechanism consisting of phase adaption, frequency adaption, and amplitude adaption is designed for mapping the frequency spectrum of the current sliding window to that of the forecasting interval. The multilayer neural networks conduct a transformation similar to the inverse discrete Fourier transform for generating a periodic feature forecast. Synthetic data and real-world data with different periodic characteristics are used to evaluate the effectiveness of the proposed model. The experimental results indicate that the ATFN has promising performance and strong adaptability for long-term time-series forecasting.