Semisupervised charting for spectral multimodal manifold learning and alignment

Abstract

For one given scene, multimodal data are acquired from multiple sensors. They share some similarities across the sensor types (redundant part of the information, also called coupling part) and they also provide modality-specific information (dissimilarities across the sensors, also called decoupling part). Additional critical knowledge about the scene can hence be extracted, which is not extractable from each modality alone. For the processing of multimodal data, we propose in this paper a model to simultaneously learn the underlying low-dimensional manifold in each modality, and locally align these manifolds across different modalities. For each pair of modalities we first build a common manifold that represents the corresponding (redundant) part of information, ignoring non-corresponding (modality specific) parts. We propose a semi-supervised learning model, using a limited amount of prior knowledge about the coupling and decoupling components of the different modalities. We propose a localized version of Laplacian eigenmaps technique specifically designed to handle multimodal manifold learning, in which the ideas of local patching of the manifolds, also known as manifold charting, is combined with the joint spectral analysis of the graph Laplacians of the different modalities. The limited given supervised information is then extending on the manifold of each modality. The idea of functional mapping is finally used to align the different manifolds across modalities. The evaluation of the proposed model using synthetic and real-world multimodal problems shows promising results, compared to several related techniques.