Cross Spectral Disparity Estimation From VIS and NIR Paired Images Using Disentangled Representation and Reversible Neural Networks

Abstract

There exists spectral gap between visible (VIS) and near infrared (NIR) images causing dissimilar intensity according to reflection property of objects and materials. Therefore, it has a limit of applying traditional stereo matching to cross spectral disparity estimation. In this paper, we propose cross spectral disparity estimation from VIS and NIR paired images using disentangled representation and reversible neural networks. We build a supervised learning framework based on reversible blocks to extract scene features robust against the spectral gap. Reversible blocks decompose features into scene and style components to bridge the spectral gap between VIS and NIR images. We perform stereo matching on the scene component to get an initial disparity map by a 3D convolutional neural network. To generate clear edges in the disparity map, we use a semantic segmentation network as auxiliary information to refine the initial disparity map. Besides, to consider the lack of the ground truth, we synthesize reference disparity maps with guided image filtering. Experimental results demonstrate that the proposed method achieves accurate edges in disparity along object boundaries and outperforms the state-of-the-art methods in both visual comparison and quantitative measurements.