CRVC-GAN: combining cross-scale fusion and recursion for video colorization adversarial generative networks

Abstract

We proposed an innovative end-to-end generative adversarial network structure (CRVC-GAN) for video colorization, which is dedicated to solving the temporal consistency and coloring quality problems in the video coloring domain. For the temporal consistency problem, we utilize the network structure of recurrent neural network (RNN) because the model shape of RNN does not change with the input length and the calculation takes into account the history information to ensure the degree of correlation between successive frames. This better ensures the continuity between frames, so the consecutive frames get good temporal consistency in coloring. For the coloring quality problem, we utilize the UNet++ coding method to decode the video frames, integrating different levels of features to improve accuracy, and then enhance the object boundary information in the video frames, thus improving the coloring quality of grayscale video frames. In addition, we innovatively utilize the style transformation loss function in terms of images for model training to ensure temporal consistency and coloring quality by exploiting the slight difference in style variation from frame to frame. The experimental results show that, compared with the existing methods and through its own designed balance index, the colorization and temporary consistency balance index, CRVC-GAN obtained certain advantages on the DAVIS and Videvo public datasets with the results of 0.56176 and 0.74286, respectively, and achieved a better balance between time consistency and color quality.