Person re-identification from virtuality to reality via modality invariant adversarial mechanism

Abstract

Person re-identification based on multi-style images helps in crime scene investigation, where only a virtual image (sketch or portrait) of the suspect is available for retrieving possible identities. However, due to the modality gap between multi-style images, standard model of person re-identification cannot achieve satisfactory performance when directly applied to match the virtual images with the real photographs. To address this problem, we propose a modality invariant adversarial mechanism (MIAM) to remove the modality gap between multi-style images. Specifically, the MIAN consists of two parts: a space transformation module to transfer the multi-style person images to a modality-invariant space, and an adversarial learning module “played” between the category classifier and modality classifier to steer the representation learning. The modality classifier discriminates between the real and virtual images while the category classifier predicts the identities of the input transformed images. We explore the space transformation for data augmentation to further bridge the modality gap and facilitate the performance. Furthermore, we build two new datasets for the multi-style Re-ID to evaluate the performance. Extensive experimental results demonstrate the effectiveness of the proposed method on improving the performance against the existing feature learning networks. Further comparison results conducted on different modules in MIAM show that our approach is of favorable generalization ability on alleviating the modality gap to improve the multi-style Re-ID.