Dual-stream spatio-temporal decoupling network for video deblurring

Abstract

It is very important to obtain spatio-temporal information in video deblurring based on deep learning. The existing methods usually jointly learn the spatio-temporal information of blurred videos through single-stream networks, which inevitably limit spatio-temporal information learning and video deblurring performance of networks. Therefore, we propose a dual-stream spatio-temporal decoupling network (STDN), which can learn the spatio-temporal information of blurred videos more flexibly and efficiently with the decoupled temporal stream and spatial stream, for solving this problem. Firstly, in the temporal stream of STDN, we propose a video deblurring pipeline, that is motion compensation plus 3D CNNs, for solving the drawback of 3D CNNs that its receptive field cannot effectively cover the same but misplaced contents of different frames. Thus, the temporal stream can aggregate temporal information of frame sequences and handle inter-frame misalignments more effectively. Specifically, we design a novel deformable convolution compensation module (DCCM) to achieve motion compensation of this pipeline more accurately. Then, we develop a 3DConv module optimized by the designed temporal, spatial, and channel decoupling attention block, named the CTS, to achieve 3D CNNs of this pipeline. Secondly, we design a spatial stream in which two types of wide-activation residual modules are stacked, for learning more spatial features of the central frame to supplement the temporal stream. Finally, extensive experiments on the baseline datasets demonstrate that the proposed STDN has better performance than the latest methods. Remarkably, using the proposed temporal stream alone already can achieve competitive video deblurring performance than the existing methods.