The continuous memory: A neural network with ordinary differential equations for continuous-time series analysis

Abstract

Continuous-time series analysis has garnered significant research interest due to its extensive applications; however, it remains a challenging endeavor. In recent years, deep learning methods have achieved remarkable success in tasks such as time series classification and forecasting. Nevertheless, the inherent continuity of time series data has not been fully addressed, presenting a persistent obstacle to performance. Recent studies have highlighted a natural similarity between differential equations and continuous-time series, as both inherently encapsulate the concept of continuity. However, several critical issues with differential equations hinder their direct applicability to time series analysis. These issues include the elusive nature of analytical solutions, the indirect observability of the effects of differential equation parameters, and the dependence of numerical solutions on initial conditions. In this context, we propose the integration of differential equations into neural networks to serve as the continuous memory of the model. This integration imparts a continuous nature to the model, resulting in the development of an efficient deep learning architecture known as Long Short Deep Memory (LSDM). Furthermore, we analyze the characteristics of differential equations when employed as the memory component in neural networks, which leads to the proposal of a novel pre-training approach that incorporates an innate memory mechanism into these networks. Additionally, we utilize LSDM to construct a stacked architecture for processing very long time series data. The proposed model can naturally accommodate arbitrary time gaps between observations, thereby enhancing its effectiveness and suitability for continuous time series analysis tasks. Extensive experiments conducted on various real-world datasets demonstrate that the proposed model outperforms existing methods, providing a new solution to the challenges associated with continuous time series analysis.