Facial feature embedded CycleGAN for VIS–NIR translation

Abstract

Visible and near-infrared (VIS–NIR) heterogeneous face recognition remains a challenging task due to distinctions between spectral components of two modalities and insufficiently pairwise VIS–NIR data. Inspired by the cycle-consistent generative adversarial network (CycleGAN), this paper proposes a facial feature embedded CycleGAN to translate between VIS and NIR face images, aiming to enable the distributions of translated (fake) images to be similar as those of true images. To learn the particular feature of NIR or VIS domain while preserving common facial representation between VIS and NIR domains, a facial feature extractor (FFE), tailored specifically for extracting effective feature from face images, is embedded in the generator of original CycleGAN. For implementing the FFE, we use the MobileFaceNet which is pre-trained on a VIS face database. The domain-invariant feature learning is enhanced by proposing a new pixel consistency loss. Additionally, we establish a new WHU VIS–NIR database including varies in face rotation and expressions to enrich the insufficient training data. Moreover, experiments on the well-known Oulu-CASIA NIR–VIS database and our WHU VIS–NIR database validate the potential benefit of the proposed FFE-based CycleGAN (FFE-CycleGAN). In particular, we achieve 96.5% accuracy on Oulu-CASIA and 98.9% accuracy on WHU VIS–NIR.