Local self-similarity as a dense stereo correspondence measure for themal-visible video registration

Abstract

The robustness of Mutual Information (MI), the most used multimodal dense stereo correspondence measure, is restricted by the size of the matching windows. However, obtaining the appropriately sized MI windows for matching thermal-visible pair of images of multiple people with various poses, clothes, distances to cameras, and different levels of occlusions is quite challenging. In this paper, we propose local self-similarity (LSS) as a multimodal dense stereo correspondence measure. We integrated LSS as a similarity metric with a disparity voting registration method to demonstrate the suitability of LSS for a visible-thermal stereo registration method. We have analyzed comparatively LSS and MI as multimodal correspondence measures and discussed LSS advantages compared to MI. We have also tested our LSS-based registration method in several indoor videos of multiple people and shown that our registration method outperforms the most recent MI-based registration method in the state-of-the-art.