Transformer with Sparse Attention Mechanism for Industrial Time Series Forecasting

Abstract

Long sequence time-series forecasting has become a very important task in industrial data analysis. Recently, the Transformer model has been widely used in sequence processing tasks. However, because industrial time series data are generally long and mixed with abnormal data, conventional Transformer model may extract irrelevant information in the context, resulting in poor forecasting. In this paper, we present Transformer with a Sparse Attention Mechanism (SAM) which can ensure local context be better integrated into attention mechanism. Inspired by the gating mechanism of LSTM, the most interesting part of sequence information are retained and the rest of the unimportant information are filtered. More attention can be focused on the factors that contribute most to the forecasting value of the sequence through this method. This method can efficiently capture long-range dependency between output and input. Furthermore, we leverage STL (Seasonal and Trend decomposition using Loess) model and IQR (Interquartile Range) method to address the outlier data. By applying this model to real-world datasets, our method achieves significant performance improvements over other methods.