NPDN-3D: A 3D neural partial differential network for spatiotemporal prediction

Abstract

Neural network-based methods have been widely applied to spatiotemporal prediction tasks, such as video prediction and weather forecasting. However, most existing works are designed for prediction in 2D space, and 3D prediction has not been extensively studied. In this paper, we propose to leverage 3D partial differential equations (PDEs) for spatiotemporal prediction in 3D space, and further develop a novel 3D neural partial differential network. This is inspired by that 3D PDEs can model both horizontal and vertical information interactions by various partial derivatives. Moreover, they can also formulate physical knowledge by equations. To integrate 3D PDEs in neural networks, we first develop the theory of approximating 3D partial derivatives by 3D convolutions, and further present an effective strategy to utilize the theory in practice. Then based on the theory and strategy, we propose a novel 3D Neural Partial Differential Network for prediction, named NPDN-3D. Specifically, NPDN-3D consists of two pivotal modules: (1) a neural partial differential module for capturing low-order spatiotemporal dynamics. This module is the key for prediction, where the dynamics are formulated by commonly-used low-order 3D PDEs. (2) A residual module for capturing the remaining non-low-order dynamics. This module performs as an extensible plug-in to enhance the expressiveness of our model. Extensive experiments on two simulated datasets and two real datasets show that our method not only achieves better prediction but also learns the correct PDEs.