Image-to-image translation for improvement of synthetic thermal infrared training data using generative adversarial networks

Abstract

Training data is an essential ingredient within supervised learning, but time consuming, expensive and for some applications impossible to acquire. A possible solution is to use synthetic training data. However, the domain shift of synthetic data makes it challenging to obtain good results when used as training data for deep learning models. It is therefore of interest to refine synthetic data, e.g. using image-to-image translation, to improve results. The aim of this work is to compare different methods to do image-to-image translation of synthetic training data of thermal IR-images using generative adversarial networks (GANs). Translation is done both using synthetic thermal IR-images alone, as well as including pixelwise depth and/or semantic information. To evaluate, we propose a new measure based on the Frechet Inception Distance, adapted to work for thermal IR-images. We show that by adapting a GAN model to also include corresponding pixelwise depth data to each synthetic IR-image, the performance is improved compared to using only IR-images.