Hierarchical Color Fusion Network (HCFN): Enhancing exemplar-based video colorization

Abstract

We conducted a study on exemplar-based colorization for grayscale videos, aiming to enhance visual perception by transforming them into vibrant videos with plausible colors. Despite promising results from existing approaches, effectively transferring colors from a reference image to grayscale video frames while maintaining temporal consistency between frames remains a challenge. To tackle this challenge, we designed a Hierarchical Color Fusion Network (HCFN). For each grayscale video frame, HCFN initially uses global and local attention mechanisms to calculate pixel-level similarity with the reference image and spatio-temporal correspondence with its neighboring frame, respectively. Based on these relationships, the pixel-level color and spatio-temporal color of the grayscale video frame are generated. Additionally, the tone-based color for the given frame is obtained based on the color distribution of the reference image. Finally, the ambiguity among these three types of colors is eliminated through the proposed hierarchical fusion mechanism, and the final color of the grayscale video frame is produced. Experimental results on public databases show that our method outperforms state-of-the-art methods in visual quality, realism and temporal consistency by a large margin. The source code and pre-trained checkpoints for HCFN is publicly available at https://github.com/wangyins/HCFN.