ClawGAN: Claw connection-based generative adversarial networks for facial image translation in thermal to RGB visible light

Abstract

Thermal cameras work well in harsh environments, but the quality of infrared images is not as high as visible light. Thermal to visible image translation can get rid of the image modal differences caused by various spectral characteristics. Nowadays, Generative Adversarial Network (GAN) can transform the images from one domain to another domain, but the generated images are still in a single channel in case of facial thermal to visible. In this paper, we propose a claw connection-based generative adversarial networks framework named ClawGAN for the facial thermal images to RGB visible images translation. We proposed the mismatch metric (MM) to measure the mapping relationship of paired images and use template matching to reduce MM of the dataset. Based on the CycleGAN framework, the synthesized loss and the generative reconstructed loss are added to the adversarial loss and the cycle-consistency loss to form a new objective function. And a claw-connected network is invoked to replace the U-net network structure of the generator for more feature preservation. The model is judged from subjective evaluation and objective evaluation based on image quality metrics such as PSNR (Peak Signal to Noise Ratio), SSIM (Structural Similarity), FID (Fréchet inception distance), and face recognition accuracy. We divided the open datasets into bright light and dark light to research the effect of illumination. The experiments show that the proposed method has the lowest FID and the highest face recognition accuracy compared to the state-of-the-art methods. The proposed ClawGAN retains the structural features of thermal images while not only enhancing the quality of images but also effectively improving the observability of image translation results in both bright and dark light. The code is available at https://github.com/Luoyi3819/ClawGAN.