Neighbor-consistent multi-modal canonical correlations for feature fusion

Abstract

Locality neighbor relationships of raw high-dimensional data are usually utilized in multi-modal feature fusion methods, and intrinsic locality neighbor relationships hidden in the raw data are beneficial to the discriminative power of fused low-dimensional features. However, since the raw data contains a lot of redundant information and noises, the neighbor relationships will deviate from intrinsic neighbor relationships and the deviation can weaken the discriminative power of the fused features. Aiming at this issue, we construct cross-modal neighbor-consistent scatters of all the modalities by explicitly embedding the neighbor complementarity of different modalities. Then we constrain the scatters in the multi-modal correlation analysis framework and further develop a novel neighbor-consistent correlation feature fusion method, i.e. neighbor-consistent multi-modal canonical correlations (NcMCC). The fused correlation features of our method preserve the intrinsic neighbor relationships with cross-modal neighbor complementarity as many as possible and possess the well discriminative power. Extensive experimental results on several categories of images such as thermal images have demonstrated the effectiveness and robustness of our method in image recognition.