Abstract
During the past decade, deep learning has represented the biggest research trend in the field of machine learning, which provides new powerful tools to interrogate high dimensional time series data in a way that has not been possible before. With the recent success in natural language processing, one would expect widespread adaptation to problems like time series forecasting and classification. After all, both involve processing sequential data. However, to this point, research on their adaptation to time series problems has remained limited. Recently, a multi-step time-stepping scheme without the need of direct access to temporal gradients has been proposed, which can accurately identify nonlinear dynamical systems from time series data. In this paper, we combined an attention mechanism with a deep model in a multi-step time-stepping scheme to perform nonlinear system identification and forecasting tasks. To our knowledge, this is the first paper to use attention based models to deal with nonlinear system identification and forecasting problems. The attention weights on rows select those variables that are helpful for forecasting, which can enhance the information across multiple time steps to capture temporal information. The experiment results indicate that the attention based model in a multi-step time-stepping scheme has better identification and prediction performance for nonlinear time series identification and forecasting problems.
Highlights
The physical laws extracted from experimental data play a critical role in many science and engineering applications, and time series data collected from experiments often obey unknown governing equations
From the fluid flow behind a cylinder to the movement of the pendulum under the influence of gravity, the mathematical models of dynamics derived from the observed data have yielded a set of methods that aim to analyze the current state of system depending on the past and to forecast the possible state in the future
More research efforts were devoted to black-box approaches that avoid the choice of the basis function to perform the system identification tasks of directly observed data
Summary
The physical laws extracted from experimental data play a critical role in many science and engineering applications, and time series data collected from experiments often obey unknown governing equations. Attention models have recently powered significant progress in natural language processing (NLP).16 With this recent success in NLP, one would expect widespread adaptation to problems like time series forecasting and classification. There are two reasons to choose the CLDNN frameworks: First, computing the local features from the data has to depend on the CNN layers; second, both CNNs and LSTM have shown improvements over DNNs across a wide variety of deep learning tasks. By comparing multistep DNN, multi-step, and attention-based multi-step CLDNN, we validate the improvement induced by the attention mechanism in several benchmark experiments The remainder of this manuscript is structured as follows: In Sec. II, we propose our model architecture and explain how to combine the attention module and CLDNN frame in the multi-step timestepping schemes.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
