Abstract
In multivariate time series modeling, it is necessary to capture short-term mutation and long-term dependence information simultaneously. However, mechanism which can capture short-term change is difficult to be used to grasp long-term dependence information, and vice versa. In order to capture both short-term mutation and long-term dependence information in the same model, this paper proposed a dual-staged attention mechanism based on conversion-gated Long Short Term Memory network(DA-CG-LSTM). Hyperbolic tangent function is introduced into the input-gate and the forget-gate of Long Short Term Memory network(LSTM), which improves the ability of the network to extract the short-term mutation information. Further, dual-staged attention mechanism is added in the network, which includes input attention and temporal attention. Input attention adaptively extracts the feature relations of exogenous sequences, and temporal attention selects the relevant hidden layer states across all the time steps. Experiments on air quality and traffic flow time series data show that the proposed network has lower average absolute error, average absolute percentage error and root mean square error by more than 50% compared with Dual-staged Attention Recurrent Neural Network(DA-RNN) and Transformation-gated LSTM(TG-LSTM).
Highlights
Time series prediction algorithms have been widely applied in many areas, such as environmental temperature[1], financial stock[2] and traffic condition data prediction[3]
Traditional models[4]-[8] have shown their effectiveness for various real world applications in multivariate time series prediction[9], they cannot model nonlinear relationships and do not differentiate among the exogenous input terms
MODEL STRUCTURE In order to effectively address issues of capturing short-term mutation and processing long-term dependence information in the same model, this paper proposes a dual-staged attention mechanism based on CONVERSION-GATED LSTM (CG-LSTM)
Summary
Time series prediction algorithms have been widely applied in many areas, such as environmental temperature[1], financial stock[2] and traffic condition data prediction[3]. Temporal attention was combined with decoder network to select relevant encoder hidden states across all time steps, which can effectively capture long-term dependent information of target sequence. B. MODEL STRUCTURE In order to effectively address issues of capturing short-term mutation and processing long-term dependence information in the same model, this paper proposes a dual-staged attention mechanism based on CG-LSTM. Before convergence of conversion-gated LSTM, multiple backpropagation calculations are carried out to learn optimal parameters of loss function, and partial derivative of conversion gate makes value range of gradient data stream constantly change At this time, short-term mutation information will correspond to the most significant interval of range change, capturing mutation law well.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
