Improving Optical Flow Inference for Video Colorization

Abstract

Recent video colorization methods optimize correspondence estimation and information propagation in an end-to-end manner. However, they usually suffer from loss of fidelity due to the inaccurate inference of correspondence measurement. In this paper, we propose a post-training optimization (PTO) strategy to refine correspondence measurement in the end-to-end optimized framework. The proposed PTO strategy introduces a pseudo loss function to well approximate the target loss and guide the direction of updates. We further develop a video colorization method that incorporates PTO and optical flow to guarantee high-fidelity colorized frames in theory. Experimental results demonstrate that the proposed method achieves state-of-the-art PSNR performance in video colorization on the DAVIS dataset and common test sequences for video coding. Furthermore, the proposed method can be employed into video compression and achieves competitive rate-distortion performance with the recent High Efficiency Video Coding (HEVC) standard.