Spatial-temporal 3D dependency matching with self-supervised deep learning for monocular visual sensing

Abstract

Monocular visual sensing is the task of using a camera to estimate the scene depth, optical flow and camera pose. In this paper, we propose a spatial–temporal 3D dependency matching approach that enforces the robustness of continuous frames matching for monocular visual sensing. 3D structure and warped depth based geometry backpropagation are used to encourage jointly learning the view depth, optical flow and camera pose employing a novel self-supervised neural network from monocular sequences. We designed two different iterative convolutional prediction sub-networks, where the optical flow obtained by depth and camera pose is iteratively used for depth prediction. A virtual frame method is proposed to optimize the optical flow of moving objects. The salient feature of the proposed learning framework is completely unsupervised, requiring only consecutive monocular images for training and testing. Evaluation on publicly benchmark datasets shows that our unsupervised learning model significantly outperforms previous methods and achieves better performance compared with previously unsupervised manners and achieves comparable results with supervised ones.