Private and common feature learning with adversarial network for RGBD object classification

Abstract

A key issue in RGBD classification is how to fuse the RGB and depth modalities. A popular way is to extract the private features of unimodality and the common features between the two modalities. Most of them use low order algebraic metrics to find the common part of the RGB and depth signals. In this paper, adversarial network is used to learn the common features between the RGB and depth modalities. A modality discriminator is designed to compete with the feature encoder so as to generate the modality-invariant information, which can be regarded as the common features. With this concept, we present a Multi-Modal Feature Learning algorithm with Adversarial Network (MMFLAN) to decouple the RGBD signals and obtain the fused features. Comprehensive experiments based on three datasets are used to evaluate the effectiveness and robustness of our MMFLAN.