Abstract

Abstract. Realistic 3D models with textures representing thermal emission of the object are widely used in such fields as dynamic scene analysis, autonomous driving, and video surveillance. Structure from Motion (SfM) methods provide a robust approach for the generation of textured 3D models in the visible range. Still, automatic generation of 3D models from the infrared imagery is challenging due to an absence of the feature points and low sensor resolution. Recent advances in Generative Adversarial Networks (GAN) have proved that they can perform complex image-to-image transformations such as a transformation of day to night and generation of imagery in a different spectral range. In this paper, we propose a novel method for generation of realistic 3D models with thermal textures using the SfM pipeline and GAN. The proposed method uses visible range images as an input. The images are processed in two ways. Firstly, they are used for point matching and dense point cloud generation. Secondly, the images are fed into a GAN that performs the transformation from the visible range to the thermal range. We evaluate the proposed method using real infrared imagery captured with a FLIR ONE PRO camera. We generated a dataset with 2000 pairs of real images captured in thermal and visible range. The dataset is used to train the GAN network and to generate 3D models using SfM. The evaluation of the generated 3D models and infrared textures proved that they are similar to the ground truth model in both thermal emissivity and geometrical shape.

Highlights

  • Thermal emission of objects captured by an infrared camera provides a whole new way for scene analysis

  • We propose a novel method for generation of realistic 3D models with thermal textures using the Structure from Motion (SfM) pipeline and

  • We present a novel iterative training method for a multimodal Generative Adversarial Networks (GAN) that overcomes most of the problems of thermal image synthesis

Read more

Summary

Introduction

Thermal emission of objects captured by an infrared camera provides a whole new way for scene analysis. A large training dataset is required to develop and train effective algorithms for processing thermal images. 3D object models with realistic textures have become one of the main instruments for the creation of extensive image datasets with accurate ground truth annotations. Structure from Motion (SfM) algorithms provide a fast and robust approach for a textured 3D model generation. For most objects, a direct 3D reconstruction using SfM and infrared range images is challenging (Hajebi and Zelek, 2008, Yamaguchi et al, 2017)

Methods
Results
Conclusion
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call