Pose aligned modality-invariant feature learning for NIR–VIS heterogeneous face recognition

Abstract

Both pose disparities and modality differences with various modalities are significant difficulties that have an impact on recognition accuracy in heterogeneous face recognition. In this paper, we propose the pose aligned modality-invariant feature learning (PAMFL) method for NIR–VIS face recognition. This method disentangles the processing of the face pose and modality into independent stages. In the first phase, we construct the face pose alignment module (PAM). The built StyleGAN2-based generator incorporates pose estimation and feature mapping structures to alter the face shape in accordance with pose yaw angle instructions, eliminating the face pose misalignment. In the second phase, we build the modality-invariant feature learning module (MFLM). Modality-specific feature representations are learned using a pseudo-Siamese network in the shallow layer of the network, while modality-invariant feature representations are learned using a parameter sharing layer embedded in the deeper layer of the network. This module preserves all modality-invariant features while minimizing cross-modality variation. Finally, comparative experiments on BUAA VisNir, CASIA NIR–VIS 2.0 and Oulu CASIA NIR–VIS datasets validate that the proposed PAMFL shows advanced performance in overcoming face pose misalignment and improving heterogeneous face recognition accuracy.