Adaptive Spatio-Temporal Convolutional Network for Video Deblurring

Abstract

Video deblurring is a challenging task due to the spatially variant blur caused by camera shake, object motions, and depth variations, etc. However, for the blurred area in the current video frame, the corresponding pixels of its neighboring video frames are often clear. Based on this observation, we propose an Adaptive Spatio-Temporal Convolutional Network (ASTCN) to compensate for blurry pixels in the current frame by using clear pixels in adjacent frames. In order to use the spatial information of adjacent frames in the current frame, the video frames must be aligned first. Existing methods usually estimate optical flow in the blurry video to align consecutive frames. However, they tend to generate artifacts when the estimated optical flow is not accurate. In order to overcome the limitations of optical flow estimation, we use deformable convolution in ASTCN to complete multi-scale adjacent frame alignment at the feature level. Secondly, we propose an adaptive spatio-temporal feature fusion module based on dynamic filters, which uses the features of the clear regions of adjacent frames to perform adaptive feature transformation on the intermediate frame to remove the blur. Extensive experimental results show that the proposed algorithm has shown superior performance on the benchmark datasets as well as real-world videos.