Optical flow correction network for repairing video

Abstract

Some areas in the video may be defective due to storage medium damage or data transmission loss. The defective area can be repaired utilizing the optical flow since the optical flow represents the motion information of the pixel and contains the information redundancy properties of the video. However, it is a challenge for current algorithms to directly calculate the correct optical flow in the defective area. Therefore, an optical flow correction network (OFC-Net) is proposed to solve this problem. Firstly, as the basic framework, OFC-Net employs a lightweight U-shaped structure to gradually “repair” the broken optical flow map based on the known information in the defective neighborhood. And then, the activation function that conforms to the property of optical flow is used to accelerate the model’s convergence during training, while the simplified loss function is adopted to evaluate the similarity of the optical flow map from a global perspective to improve the prediction accuracy. As a result, the corrected optical flow output at the masked position can be obtained. Secondly, the optical flow corrected by OFC-Net can guide locating the pixels in nearby frames to repair the defective region. Optical flow correction and video repair experiments were conducted on MPI Sintel, DAVIS datasets, and real movie clips. The experimental results show that the corrected optical flow map is accurate without bringing artifacts under a wide range of scenarios and the arbitrary irregular mask shapes, which demonstrate the network’s strong generalization capabilities. The average PSNR and SSIM of repaired video frames are greater than 33.67 and 0.986, respectively, outperforming the results from the traditional method PatchMatch and deep learning-based synthetic pixel methods.