Thermal-visible stereo matching at night based on Multi-Modal Autoencoder

Abstract

Compared to RGB cameras, thermal sensors are more reliable at night. However, in the stereo matching task, it is difficult to estimate reliable depth simply depending on the binocular thermal cameras, due to their monochrome and low-contrast images. This paper proposed a thermal-visible stereo matching method based on multi-modal autoencoder (MANet), the key to which is to extract the modal-invariant features between thermal and visible images, and meanwhile, retain the characteristic features in terms of the optical properties. Besides, we designed a cross-reconstruction constraint to improve the scene feature extraction capability of MANet. In the stereo matching process, we analyze the thermal and visible properties and adopt optical attention mechanism to improve night-scene stereo matching. We established a stereo matching system with thermal-visible camera based on LiDAR calibration, and provided the dataset for evaluation. The experiment demonstrates that the proposed MANet can effectively extract the modal-invariant features and can be applied on thermal-visible stereo matching.