Cross-spectral stereo matching for facial disparity estimation in the dark

Abstract

Numerous applications on human faces hinge on depth information. Often, facial stereo matching provides an opportunity to estimate disparity without active projectors. However, existing algorithms are less effective at night due to unclear texture and severe noises in RGB images. In this paper, we address this problem by estimating facial disparity maps from NIR-RGB pairs. We develop a neural network composed of a multi-spectral transfer network (MSTN) and a disparity estimation network (DEN). MSTN is used to produce a pseudo-NIR image aligned with the RGB view using a spatially weighted sum on the NIR one by a kernel prediction network (KPN). As the pseudo-NIR and the NIR images share the same appearance, the facial disparity map is predicted by the proposed DEN with the same-spectral stereo pair. The whole network can be trained in an end-to-end manner and the experimental results demonstrate that it performs favorably against state-of-the-art algorithms on both synthetic and real data.