Interactive image colorization method via dual local self-attention

Abstract

The interactive colorization allows users to add a desired color to any location in a grayscale image to obtain the desired color image. While transformers can capture larger receptive fields compared to convolutional neural networks, their computational complexity in image colorization is too high. As a solution, this paper introduces a novel image colorization framework for processing image colorization assignments. In our framework, the dual local self-attention mechanism model is formulated as a solution for reducing the computational complexity associated with global self-attention mechanisms, where the improvement is to combine shift window local self-attention and feature space local self-attention. Both local spatial connections and distant relations are captured to enhance the local quality for the reconstruction. Prior to computing feature space local self-attention, a brightness similarity metric is introduced to cluster the interested area to different local regions. Aiming for real-time inference, the DUpsampling is adopted for signal reconstruction. Specifically, the lightweight convolutional layer is employed before the DUpsampling to mitigate artifacts. Our experimental results on multiple datasets demonstrate that our proposed method outperforms existing interactive colorization methods while having lower model parameters and computational complexity than other Transformer-based models.