MUGAN: Thermal Infrared Image Colorization Using Mixed-Skipping UNet and Generative Adversarial Network

Abstract

Common cameras cannot capture high quality image in the night or extreme weather conditions that without enough light, while thermal infrared(TIR) cameras are not limited in this situation. Hence, TIR imaging technique is widely used in military, surveillance, nighttime traffic and other scenarios. However, TIR images are monochromatic, and the majority of details of such images are lost, which are difficult for human or computer system to analyze. Translating TIR images into visible images is beneficial to subsequent observation or further processing. Though there are some advances to realize the transformation from TIR images to color visible images, edge distortion and semantic confusion remain to be solved. Therefore, we propose a Mixed-Skipping UNet(MS-UNet) based image colorization model joint Generative Adversarial Network, which is denoted by MUGAN. Firstly, the dense skip connections of UNet++ and full-scale skip connections of UNet 3+ are combined to form the MS-UNet, which is regarded as the generator. In addition, we design a feature extraction module in MS-UNet to effectively capture the multi-scale features in source image. Then, a novel attention mechanism module is designed for decoding stage which can help decoder of MS-UNet to focus on the important information. Moreover, we explore the effect of different loss functions in TIR image colorization task, and the loss function with excellent performance is selected to further optimize the training process of the model. Extensive experiments prove the superiority of our method in the task of TIR image colorization. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/HangyingLiao/MUGAN</uri> .