VD-GAN: A Unified Framework for Joint Prototype and Representation Learning From Contaminated Single Sample per Person

Abstract

Single sample per person (SSPP) face recognition with a ${c}$ ontaminated biometric ${e}$ nrolment database (SSPP-ce FR) is an emerging practical FR problem, where the SSPP in the enrolment database is no longer standard but contaminated by nuisance facial variations such as expression, lighting, pose, and disguise. In this case, the conventional SSPP FR methods, including the patch-based and generic learning methods, will suffer from serious performance degradation. Few recent methods were proposed to tackle SSPP-ce FR by either performing prototype learning on the contaminated enrolment database or learning discriminative representations that are robust against variation. Despite that, most of these approaches can only handle a specified single variation, e.g., pose, but cannot be extended to multiple variations. To address these two limitations, we propose a novel Variation Disentangling Generative Adversarial Network (VD-GAN) to jointly perform prototype learning and representation learning in a unified framework. The proposed VD-GAN consists of an encoder-decoder structural generator and a multi-task discriminator to handle universal variations including single, multiple, and even mixed variations in practice. The generator and discriminator play an adversarial game such that the generator learns a discriminative identity representation and generates an identity-preserved prototype for each face image, while the discriminator aims to predict face identity label, distinguish real vs. fake prototype, and disentangle target variations from the learned representations. Qualitative and quantitative evaluations on various real-world face datasets containing single/multiple and mixed variations demonstrate the effectiveness of VD-GAN.